00000100:00080000:1.0:1769075170.792678:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075170.792680:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075170.792683:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2156 00000100:00080000:1.0:1769075170.792686:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075170.792690:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075170.792698:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075170.792708:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616335360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075170.792719:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075170.792726:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075170.792728:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075170.792731:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2156 00000100:00080000:1.0:1769075170.792733:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075170.792735:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075170.792738:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075170.792748:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616335488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075170.792758:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075170.792762:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075170.792763:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075170.792768:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2156 00000100:00080000:1.0:1769075170.792781:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075170.792787:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075170.792793:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075170.792810:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616335616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075170.792822:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075170.792825:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075170.792828:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075170.792832:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2156 00000100:00080000:1.0:1769075170.792858:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075170.792869:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075170.792878:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075170.792894:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616335744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075170.792904:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075170.792912:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075170.792918:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075170.792930:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2156 00000100:00080000:1.0:1769075170.792935:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075170.792943:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075170.792952:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075170.792966:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616335872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075170.792983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075170.793056:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075170.793062:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075170.793071:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2156 00000100:00080000:1.0:1769075170.793080:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075170.793089:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075170.793097:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075170.793116:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616336000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075171.496581:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075171.816546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075173.864569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075174.568768:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075174.888592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075175.912704:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075176.936548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075177.640669:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075177.960660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075178.984662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075180.712629:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075181.032908:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075182.056550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075182.056671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075183.080652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075183.784590:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075184.104542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075185.128555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075186.856639:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075189.224620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075189.928595:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075190.248542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075191.272520:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075191.272681:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075191.272690:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075191.272696:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075191.272706:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2176 00000100:00080000:1.0:1769075191.272714:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075191.272722:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075191.272757:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616338816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075191.272778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075191.272781:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075191.272782:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075191.272785:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2176 00000100:00080000:1.0:1769075191.272788:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075191.272790:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075191.272797:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075191.272808:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616338944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075191.272817:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075191.272824:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075191.272825:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075191.272828:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2176 00000100:00080000:1.0:1769075191.272831:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075191.272833:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075191.272836:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075191.272845:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616339072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075191.272852:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075191.272854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075191.272856:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075191.272858:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2176 00000100:00080000:1.0:1769075191.272867:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075191.272870:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075191.272873:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075191.272882:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616339200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075191.272890:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075191.272892:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075191.272894:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075191.272897:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2176 00000100:00080000:1.0:1769075191.272899:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075191.272901:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075191.272904:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075191.272913:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616339328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075191.272925:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075191.272932:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075191.272935:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075191.272947:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2176 00000100:00080000:1.0:1769075191.272950:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075191.272956:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075191.272963:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075191.272974:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616339456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075191.272984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075191.272989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075191.272992:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075191.272999:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2176 00000100:00080000:1.0:1769075191.273002:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075191.273005:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075191.273009:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075191.273015:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616339584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075192.296651:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075193.000732:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075193.320599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075193.320725:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075194.344669:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075194.344728:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075195.368658:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075196.072767:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075196.392673:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075197.416663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075198.440627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075198.440712:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075199.144624:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075199.464646:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075200.488632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075201.512592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075201.512678:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075202.216690:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075202.536569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075203.560561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075204.584695:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075204.584801:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075205.288630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075205.608541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075205.608593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075206.632537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075206.632571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075207.656527:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075208.360609:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075208.680527:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075208.680611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075209.704559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075210.728576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075211.432640:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075211.752542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075212.776532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075212.776638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075213.800544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075214.504726:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075214.824537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075215.848627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075216.872564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075216.872652:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075217.576603:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075217.896553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075218.920542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075219.944594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075220.648814:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075220.968588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075223.016629:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075223.720709:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075224.040593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075226.088647:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075226.792617:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075227.112562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075228.136560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075229.160550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075229.160585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075229.864656:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075231.208596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075232.232569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075232.936598:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075233.256582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075235.304560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075235.304602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075236.008750:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075236.328655:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075237.352958:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616346880:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769075237.353100:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075237.353191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075237/real 1769075237] req@ffff995b85aca680 x1855007616346880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075237.353231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca680 x1855007616346880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075237.353349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075237.353362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075237.353377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616346880:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075237.353436:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075237.353540:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075237.353590:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075237/real 1769075237] req@ffff995b85ac9c00 x1855007616347008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075237.353645:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616347008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075237.353676:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075237.353697:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075237.353714:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075237.353764:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075237.353820:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075237.353864:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075237/real 1769075237] req@ffff995b85ac8380 x1855007616347136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075237.353911:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616347136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075237.353931:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075237.353940:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075237.353952:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075237.353967:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347264:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075237.354000:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075237.354014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075237/real 1769075237] req@ffff995b85acb100 x1855007616347264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075237.354050:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616347264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075237.354064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075237.354070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075237.354075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075237.354108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347392:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075237.354145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075237.354164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075237/real 1769075237] req@ffff995b85acaa00 x1855007616347392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075237.354196:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616347392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075237.354204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075237.354208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075237.354211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075237.354217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616347520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075237.354238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075237.354248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075237/real 1769075237] req@ffff995b85ac8700 x1855007616347520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075237.354263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8700 x1855007616347520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075237.354284:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075237.354306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075237.354311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616347520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075237.354320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616347648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075237.354350:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075237.354367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075237/real 1769075237] req@ffff995b85aca300 x1855007616347648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075237.354384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616347648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075237.354393:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075237.354396:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075237.354399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616347648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075238.376564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075239.080668:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075239.400547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075240.424596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075241.448597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075242.152628:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075242.472538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075242.472691:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616347776:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769075242.472804:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075242.472854:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075242/real 1769075242] req@ffff995b85acbb80 x1855007616347776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075242.472884:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616347776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075242.472906:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075242.472914:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075242.472926:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616347776:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075242.472961:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075242.472996:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075242.473009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075242/real 1769075242] req@ffff995b85ac9880 x1855007616347904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075242.473018:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616347904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075242.473024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075242.473027:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075242.473029:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616347904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075242.473033:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075242.473044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075242.473050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075242/real 1769075242] req@ffff995b85ac9500 x1855007616348032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075242.473063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9500 x1855007616348032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075242.473069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075242.473071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075242.473072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075242.473076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075242.473087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075242.473101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075242/real 1769075242] req@ffff995b85acad80 x1855007616348160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075242.473109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acad80 x1855007616348160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075242.473115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075242.473117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075242.473119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075242.473122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348288:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075242.473134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075242.473145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075242/real 1769075242] req@ffff995b85acb800 x1855007616348288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075242.473158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616348288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075242.473172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075242.473176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075242.473180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075242.473186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616348416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075242.473206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075242.473215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075242/real 1769075242] req@ffff995b85ac8a80 x1855007616348416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075242.473223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616348416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075242.473228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075242.473230:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075242.473232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616348416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075242.473246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616348544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075242.473257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075242.473266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075242/real 1769075242] req@ffff995b85ac9f80 x1855007616348544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075242.473278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9f80 x1855007616348544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075242.473287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075242.473290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075242.473294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616348544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075243.496605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075244.520598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075245.224600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075245.544617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075246.568547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075247.592590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075247.592898:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616348672:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769075247.592975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075247.593026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075247/real 1769075247] req@ffff995b83505500 x1855007616348672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075247.593050:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505500 x1855007616348672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075247.593060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075247.593064:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075247.593067:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616348672:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075247.593074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075247.593092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075247.593100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075247/real 1769075247] req@ffff995b83506300 x1855007616348800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075247.593109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616348800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075247.593114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075247.593117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075247.593119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075247.593123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075247.593133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075247.593139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075247/real 1769075247] req@ffff995b83507100 x1855007616348928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075247.593146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616348928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075247.593152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075247.593153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075247.593155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616348928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075247.593174:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349056:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075247.593184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075247.593190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075247/real 1769075247] req@ffff995b83507480 x1855007616349056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075247.593198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507480 x1855007616349056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075247.593203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075247.593205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075247.593207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075247.593210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349184:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075247.593220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075247.593226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075247/real 1769075247] req@ffff995b83504a80 x1855007616349184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075247.593233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504a80 x1855007616349184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075247.593237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075247.593239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075247.593241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075247.593244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616349312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075247.593253:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075247.593258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075247/real 1769075247] req@ffff995b83505880 x1855007616349312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075247.593265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505880 x1855007616349312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075247.593270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075247.593272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075247.593274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616349312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075247.593277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616349440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075247.593286:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075247.593291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075247/real 1769075247] req@ffff995b83504000 x1855007616349440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075247.593303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616349440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075247.593309:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075247.593310:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075247.593312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616349440:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075248.296588:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075248.616541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075249.640624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075250.664588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075251.368600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075252.712650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075252.712847:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616349568:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769075252.712946:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075252.712997:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075252/real 1769075252] req@ffff995b85ac9f80 x1855007616349568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075252.713021:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9f80 x1855007616349568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075252.713056:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075252.713067:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075252.713081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616349568:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075252.713119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075252.713144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075252.713155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075252/real 1769075252] req@ffff995b85ac8a80 x1855007616349696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075252.713164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616349696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075252.713170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075252.713172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075252.713175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075252.713179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075252.713190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075252.713196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075252/real 1769075252] req@ffff995b85acb800 x1855007616349824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075252.713204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616349824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075252.713209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075252.713211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075252.713213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075252.713217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349952:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075252.713231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075252.713240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075252/real 1769075252] req@ffff995b85acad80 x1855007616349952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075252.713252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acad80 x1855007616349952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075252.713261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075252.713274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075252.713278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616349952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075252.713285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350080:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075252.713303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075252.713311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075252/real 1769075252] req@ffff995b85ac9500 x1855007616350080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075252.713319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9500 x1855007616350080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075252.713324:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075252.713327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075252.713329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075252.713332:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616350208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075252.713343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075252.713349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075252/real 1769075252] req@ffff995b85ac9880 x1855007616350208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075252.713355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616350208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075252.713360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075252.713362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075252.713364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616350208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075252.713368:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616350336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075252.713377:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075252.713383:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075252/real 1769075252] req@ffff995b85acbb80 x1855007616350336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075252.713389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616350336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075252.713395:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075252.713397:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075252.713399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616350336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075253.736550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075253.736630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075254.440666:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075254.760611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075255.784639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075256.808609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075257.512660:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075257.832596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075257.832780:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616350464:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769075257.832868:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075257.832917:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075257/real 1769075257] req@ffff995b84559180 x1855007616350464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075257.832955:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559180 x1855007616350464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075257.832983:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075257.832992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075257.833010:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616350464:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075257.833044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075257.833080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075257.833097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075257/real 1769075257] req@ffff995b84558380 x1855007616350592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075257.833113:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558380 x1855007616350592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075257.833122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075257.833126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075257.833130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075257.833136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075257.833153:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075257.833163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075257/real 1769075257] req@ffff995b8455b800 x1855007616350720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075257.833174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b800 x1855007616350720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075257.833183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075257.833186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075257.833189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075257.833195:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075257.833211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075257.833221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075257/real 1769075257] req@ffff995b8455bb80 x1855007616350848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075257.833232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616350848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075257.833239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075257.833242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075257.833245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075257.833251:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075257.833268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075257.833279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075257/real 1769075257] req@ffff995b8455aa00 x1855007616350976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075257.833290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616350976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075257.833298:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075257.833301:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075257.833305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616350976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075257.833324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616351104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075257.833344:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075257.833356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075257/real 1769075257] req@ffff995b84559500 x1855007616351104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075257.833369:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616351104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075257.833377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075257.833380:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075257.833384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616351104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075257.833390:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616351232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075257.833408:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075257.833420:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075257/real 1769075257] req@ffff995b84558a80 x1855007616351232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075257.833432:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616351232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075257.833441:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075257.833444:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075257.833447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616351232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075258.856625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075259.880686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075259.880802:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075260.584810:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075260.904634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075261.928662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075262.952695:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075262.952829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616351360:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769075262.953025:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075262.953134:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075262/real 1769075262] req@ffff995b85aca300 x1855007616351360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075262.953184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616351360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075262.953206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075262.953215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075262.953224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616351360:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075262.953269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616351488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075262.953327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075262.953364:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075262/real 1769075262] req@ffff995b85ac8700 x1855007616351488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075262.953388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8700 x1855007616351488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075262.953403:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075262.953421:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075262.953427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616351488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075262.953521:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616351616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075262.953566:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075262.953590:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075262/real 1769075262] req@ffff995b85acaa00 x1855007616351616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075262.953628:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616351616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075262.953646:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075262.953653:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075262.953660:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616351616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075262.953672:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616351744:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075262.953709:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075262.953728:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075262/real 1769075262] req@ffff995b85acb100 x1855007616351744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075262.953751:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616351744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075262.953766:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075262.953773:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075262.953780:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616351744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075262.953792:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616351872:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075262.953827:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075262.953848:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075262/real 1769075262] req@ffff995b85ac8380 x1855007616351872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075262.953871:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616351872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075262.953899:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075262.953905:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075262.953911:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616351872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075262.953924:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616352000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075262.953960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075262.953982:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075262/real 1769075262] req@ffff995b85ac9c00 x1855007616352000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075262.954006:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616352000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075262.954022:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075262.954029:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075262.954035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616352000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075262.954047:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616352128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075262.954079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075262.954100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075262/real 1769075262] req@ffff995b85aca680 x1855007616352128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075262.954124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca680 x1855007616352128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075262.954147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075262.954153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075262.954160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616352128:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075263.656728:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075263.976531:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075263.976574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075265.000571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075266.024545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075266.024625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075266.728665:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075267.048577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075267.048620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075268.072511:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075268.072732:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616352256:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075268.072857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616352384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075268.072886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616352512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075268.072910:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616352640:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075268.072937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616352768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075268.072962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616352896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075268.073053:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616353024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075269.096517:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075269.800735:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075270.120579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075271.144536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075271.144578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769075271.144766:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769075271.144803:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769075271.144819:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075271.144851:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075271.144858:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075271.144863:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075271.144873:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075271.144884:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075271.144895:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:1.0:1769075272.168531:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075272.168563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075272.872617:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075273.192576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075273.192641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075273.192672:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075274.216575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075274.216615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075275.240562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075275.240624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075275.944616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075276.264563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075277.288583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075278.312605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075279.016664:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075279.336536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075280.360514:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075281.384526:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075282.088595:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075283.432582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075284.456574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075285.160696:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075285.480540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075286.504649:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075287.528582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075287.528720:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075288.232692:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075288.552547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075289.576548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075290.600669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075291.304744:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075291.624677:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075292.648664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075293.672607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075294.376803:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075294.696646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075295.720561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075296.744644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075297.448652:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075297.768594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075298.792554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075298.792601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075299.816560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075300.520568:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075300.840533:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075303.592718:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075303.912637:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075303.912646:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075303.912652:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075303.912668:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2289 00000100:00080000:1.0:1769075303.912683:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075303.912694:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075303.912774:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5880 x1855007616358528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075303.912800:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075303.912803:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075303.912805:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075303.912809:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2289 00000100:00080000:1.0:1769075303.912812:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075303.912816:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075303.912826:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075303.912842:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616358656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075303.912853:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075303.912856:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075303.912857:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075303.912859:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2289 00000100:00080000:1.0:1769075303.912861:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075303.912863:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075303.912866:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075303.912874:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616358784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075303.912881:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075303.912884:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075303.912885:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075303.912887:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2289 00000100:00080000:1.0:1769075303.912896:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075303.912899:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075303.912903:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075303.912911:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616358912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075303.912918:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075303.912920:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075303.912922:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075303.912924:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2289 00000100:00080000:1.0:1769075303.912927:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075303.912933:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075303.912938:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075303.912944:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616359040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075303.912951:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075303.912955:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075303.912956:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075303.912959:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2289 00000100:00080000:1.0:1769075303.912961:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075303.912963:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075303.912966:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075303.912975:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7b80 x1855007616359168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075303.913014:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075303.913017:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075303.913018:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075303.913023:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2289 00000100:00080000:1.0:1769075303.913028:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075303.913034:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075303.913041:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075303.913052:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6680 x1855007616359296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075303.913304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6680 x1855007616359296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075303.913314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075303.913316:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075303.913318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616359296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075304.936533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075304.936564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075304.936578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075305.960526:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075305.960542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075306.664800:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075306.984561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075306.984572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075308.008649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075308.008649:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075309.032536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075309.032543:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075309.032549:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075309.032553:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075309.032562:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2294 00000100:00080000:1.0:1769075309.032575:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075309.032582:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075309.032616:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5180 x1855007616359424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075309.032645:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075309.032650:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075309.032653:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075309.032659:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2294 00000100:00080000:1.0:1769075309.032663:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075309.032681:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075309.032689:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616359424:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769075309.032692:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075309.032711:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6680 x1855007616359552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075309.032725:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075309.032729:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075309.032730:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075309.032733:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2294 00000100:00080000:1.0:1769075309.032739:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075309.032742:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075309.032746:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769075309.032747:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075309.032760:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7b80 x1855007616359680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075309.032768:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075309.032770:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075309.032771:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075309.032774:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2294 00000100:00080000:1.0:1769075309.032777:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075309.032781:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075309.032784:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075309.032785:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075309/real 1769075309] req@ffff995b868c5180 x1855007616359424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075309.032797:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616359808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075309.032804:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5180 x1855007616359424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075309.032817:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075309.032820:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075309.032824:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075309.032826:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769075309.032829:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075309.032831:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616359424:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075309.032839:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2294 00000100:00080000:1.0:1769075309.032845:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075309.032849:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075309.032855:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616359552:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769075309.032856:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075309.032873:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616359936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769075309.032880:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075309.032886:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075309.032890:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075309.032892:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075309.032895:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075309/real 1769075309] req@ffff995b868c6680 x1855007616359552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075309.032898:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2294 00000100:00080000:1.0:1769075309.032905:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075309.032911:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075309.032919:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075309.032925:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6680 x1855007616359552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075309.032931:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616360064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075309.032935:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075309.032943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769075309.032945:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769075309.032948:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616359552:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075309.032950:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075309.032954:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075309.032957:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616359680:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075309.032960:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2294 00000100:00080000:1.0:1769075309.032964:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075309.032967:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075309.032972:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769075309.032977:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075309.032983:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616360192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075309.032988:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075309/real 1769075309] req@ffff995b868c7b80 x1855007616359680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075309.033005:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7b80 x1855007616359680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075309.033011:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075309.033013:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075309.033015:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616359680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075309.033021:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616359808:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075309.033033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075309.033039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075309/real 1769075309] req@ffff995b868c5c00 x1855007616359808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075309.033047:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616359808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075309.033058:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075309.033060:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075309.033062:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616359808:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075309.033075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616359936:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075309.033088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075309.033100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075309/real 1769075309] req@ffff995b868c5500 x1855007616359936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075309.033112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616359936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075309.033120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075309.033125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075309.033129:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616359936:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075309.033137:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616360064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075309.033163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075309.033177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075309/real 1769075309] req@ffff995b868c7800 x1855007616360064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075309.033188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616360064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075309.033195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075309.033198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075309.033202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616360064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075309.033207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616360192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075309.033222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075309.033229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075309/real 1769075309] req@ffff995b868c4a80 x1855007616360192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075309.033238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616360192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075309.033244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075309.033248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075309.033250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616360192:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075309.736743:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075310.056556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075310.056590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075311.080670:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075311.080713:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075312.104601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075312.104685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075312.808742:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075313.128626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075313.128637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075314.152619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075314.152622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075314.152633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075314.152639:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075314.152655:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2299 00000100:00080000:1.0:1769075314.152671:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075314.152686:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075314.152760:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5880 x1855007616360320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075314.152818:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075314.152831:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075314.152833:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075314.152840:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2299 00000100:00080000:1.0:1769075314.152845:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075314.152851:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075314.152861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616360320:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769075314.152864:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075314.152882:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616360448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075314.152907:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075314.152911:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075314.152914:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075314.152919:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2299 00000100:00080000:1.0:1769075314.152923:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075314.152929:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075314.152934:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075314.152947:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616360576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075314.152968:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769075314.152969:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075314.152975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075314.152978:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075314.152984:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2299 00000100:00080000:1.0:1769075314.152989:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075314.152995:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075314.153001:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075314.153014:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616360704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075314.153020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075314/real 1769075314] req@ffff995b868c5880 x1855007616360320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075314.153036:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075314.153041:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075314.153044:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075314.153045:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5880 x1855007616360320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075314.153055:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2299 00000100:00080000:1.0:1769075314.153062:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075314.153070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075314.153076:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769075314.153079:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075314.153082:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616360320:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769075314.153089:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075314.153106:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616360832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075314.153111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616360448:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075314.153128:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075314.153132:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075314.153135:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769075314.153139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075314.153145:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2299 00000100:00080000:1.0:1769075314.153151:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769075314.153153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075314/real 1769075314] req@ffff995b868c7100 x1855007616360448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075314.153159:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075314.153169:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075314.153169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616360448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075314.153180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075314.153184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075314.153189:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616360960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075314.153189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616360448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075314.153201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616360576:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075314.153213:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075314.153217:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1769075314.153221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075314.153222:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075314.153230:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2299 00000100:00100000:3.0:1769075314.153233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075314/real 1769075314] req@ffff995b868c6300 x1855007616360576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075314.153236:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075314.153246:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075314.153250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616360576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769075314.153253:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769075314.153262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075314.153266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075314.153269:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616360576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075314.153274:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616361088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075314.153278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616360704:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075314.153300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075314.153309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075314/real 1769075314] req@ffff995b868c6d80 x1855007616360704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075314.153317:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616360704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075314.153323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075314.153326:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075314.153328:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616360704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075314.153338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616360832:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075314.153347:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075314.153353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075314/real 1769075314] req@ffff995b868c4e00 x1855007616360832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075314.153360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4e00 x1855007616360832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075314.153366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075314.153368:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075314.153370:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616360832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075314.153373:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616360960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075314.153382:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075314.153388:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075314/real 1769075314] req@ffff995b868c6a00 x1855007616360960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075314.153396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616360960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075314.153401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075314.153403:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075314.153404:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616360960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075314.153408:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616361088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075314.153417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075314.153424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075314/real 1769075314] req@ffff995b868c7480 x1855007616361088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075314.153432:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616361088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075314.153437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075314.153439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075314.153441:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616361088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075315.176609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075315.176610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075315.176650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075315.880604:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075316.200658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075316.200792:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075316.904595:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 16/16 locks to regular blp list, 1 blwis in pool 00000100:00080000:1.0:1769075316.905120:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83507480 x1855007616361216/t0(0) o103->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:17/18 lens 440/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075317.224587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075317.928586:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075318.248588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075318.248644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075319.272563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075319.272587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075319.272783:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616361344:192.168.204.132@tcp:250:kworker.0 00000800:00000100:3.0:1769075319.272875:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075319.272925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075319/real 1769075319] req@ffff995b85acbb80 x1855007616361344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075319.273006:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616361344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075319.273038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075319.273053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075319.273058:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616361344:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075319.273086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616361472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075319.273111:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075319.273121:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075319/real 1769075319] req@ffff995b85ac9880 x1855007616361472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075319.273130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616361472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075319.273136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075319.273139:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075319.273141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616361472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075319.273145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616361600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075319.273155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075319.273161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075319/real 1769075319] req@ffff995b85ac9500 x1855007616361600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075319.273175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9500 x1855007616361600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075319.273185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075319.273189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075319.273192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616361600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075319.273199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616361728:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075319.273221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075319.273232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075319/real 1769075319] req@ffff995b85acad80 x1855007616361728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075319.273243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acad80 x1855007616361728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075319.273249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075319.273253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075319.273256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616361728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075319.273263:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616361856:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075319.273283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075319.273294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075319/real 1769075319] req@ffff995b85acb800 x1855007616361856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075319.273307:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616361856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075319.273318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075319.273321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075319.273325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616361856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075319.273331:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616361984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075319.273352:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075319.273365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075319/real 1769075319] req@ffff995b85ac8a80 x1855007616361984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075319.273378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616361984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075319.273385:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075319.273388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075319.273390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616361984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075319.273395:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616362112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075319.273410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075319.273417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075319/real 1769075319] req@ffff995b85ac9f80 x1855007616362112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075319.273425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9f80 x1855007616362112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075319.273430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075319.273431:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075319.273434:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616362112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075320.296578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075320.296593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075321.000706:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075321.320538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075321.320538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075322.344560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075322.344563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075323.368619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075323.368645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075323.368705:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075324.072716:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075324.392628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075324.392667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075324.392679:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075324.392688:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075324.392707:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2310 00000100:00080000:1.0:1769075324.392727:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075324.392741:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075324.392808:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616362240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075324.392847:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075324.392853:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075324.392856:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075324.392861:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2310 00000100:00080000:1.0:1769075324.392867:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075324.392873:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075324.392887:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075324.392908:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616362368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075324.392923:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075324.392928:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075324.392930:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075324.392936:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2310 00000100:00080000:1.0:1769075324.392940:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075324.392944:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075324.392950:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075324.392957:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616362240:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075324.392968:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616362496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075324.392983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075324.392988:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075324.392990:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075324.393005:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2310 00000100:00080000:1.0:1769075324.393010:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075324.393015:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075324.393022:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075324.393041:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616362624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075324.393063:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075324.393068:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075324.393071:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075324.393075:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2310 00000100:00080000:1.0:1769075324.393079:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075324.393084:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769075324.393093:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769075324.393097:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075324.393116:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616362752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075324.393130:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075324.393134:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075324.393137:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075324.393142:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2310 00000100:00080000:1.0:1769075324.393147:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075324.393154:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075324.393161:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075324.393168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075324/real 1769075324] req@ffff995b868c4700 x1855007616362240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075324.393183:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616362880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075324.393201:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769075324.393207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616362240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075324.393210:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075324.393215:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075324.393228:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2310 00000100:00080000:1.0:1769075324.393243:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075324.393247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075324.393249:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769075324.393264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769075324.393265:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075324.393275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616362240:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075324.393285:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616363008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075324.393293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616362368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075324.393342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075324.393363:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075324/real 1769075324] req@ffff995b868c7480 x1855007616362368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075324.393378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616362368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075324.393388:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075324.393392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075324.393398:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616362368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075324.393532:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616362496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075324.393563:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075324.393584:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075324/real 1769075324] req@ffff995b868c6a00 x1855007616362496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075324.393609:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616362496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075324.393628:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075324.393634:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075324.393642:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616362496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075324.393656:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616362624:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075324.393697:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075324.393718:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075324/real 1769075324] req@ffff995b868c4e00 x1855007616362624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075324.393743:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4e00 x1855007616362624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075324.393759:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075324.393766:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075324.393772:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616362624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075324.393785:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616362752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075324.393825:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075324.393848:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075324/real 1769075324] req@ffff995b868c6d80 x1855007616362752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075324.393873:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616362752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075324.393889:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075324.393895:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075324.393901:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616362752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075324.393914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616362880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075324.393954:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075324.393978:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075324/real 1769075324] req@ffff995b868c6300 x1855007616362880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075324.393995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616362880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075324.394006:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075324.394010:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075324.394014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616362880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075324.394022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616363008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075324.394046:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075324.394059:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075324/real 1769075324] req@ffff995b868c7100 x1855007616363008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075324.394073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616363008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075324.394084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075324.394089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075324.394093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616363008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075325.416662:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075325.416727:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075325.416765:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075325.416839:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075326.440522:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075326.440604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075327.144590:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075327.464569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075327.464576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075327.464649:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075328.488571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075328.488580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075328.488611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075329.512532:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075329.512554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075329.512713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616363136:192.168.204.132@tcp:250:kworker.0 00000800:00000100:3.0:1769075329.512803:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075329.512844:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075329/real 1769075329] req@ffff995b83507100 x1855007616363136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075329.512879:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616363136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075329.512896:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075329.512905:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075329.512910:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616363136:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075329.512938:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616363264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075329.512962:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075329.512973:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075329/real 1769075329] req@ffff995b83506300 x1855007616363264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075329.512996:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616363264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075329.513002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075329.513004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075329.513006:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616363264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075329.513011:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616363392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075329.513023:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075329.513030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075329/real 1769075329] req@ffff995b83505500 x1855007616363392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075329.513037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505500 x1855007616363392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075329.513043:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075329.513045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075329.513047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616363392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075329.513050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616363520:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075329.513060:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075329.513066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075329/real 1769075329] req@ffff995b83506680 x1855007616363520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075329.513074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506680 x1855007616363520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075329.513083:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075329.513085:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075329.513086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616363520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075329.513090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616363648:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075329.513099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075329.513104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075329/real 1769075329] req@ffff995b83504380 x1855007616363648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075329.513110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504380 x1855007616363648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075329.513115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075329.513117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075329.513119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616363648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075329.513122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616363776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075329.513131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075329.513137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075329/real 1769075329] req@ffff995b83506a00 x1855007616363776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075329.513143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506a00 x1855007616363776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075329.513148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075329.513150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075329.513152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616363776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075329.513165:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616363904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075329.513175:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075329.513181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075329/real 1769075329] req@ffff995b83505c00 x1855007616363904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075329.513188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505c00 x1855007616363904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075329.513193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075329.513195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075329.513197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616363904:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075330.216590:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075330.536532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075330.536535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075330.536726:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075331.560538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075331.560538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075332.584554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075332.584555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075333.288643:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075333.608586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075333.608608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075334.632612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075334.632612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075334.632912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616364032:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075334.633104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616364160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075334.633159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616364288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075334.633307:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616364416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075334.633410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616364544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075334.633572:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616364672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075334.633640:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616364800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075335.656792:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075336.360709:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075336.680671:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075336.680696:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075336.680716:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075337.704557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075337.704592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075337.705002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075334/real 1769075337] req@ffff995b83506680 x1855007616364800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075337.705026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506680 x1855007616364800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075337.705040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075337.705045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075337.705050:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616364800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075337.705058:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075334/real 1769075337] req@ffff995b83504380 x1855007616364672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075337.705071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504380 x1855007616364672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075337.705080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075337.705083:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075337.705087:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616364672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075337.705093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075334/real 1769075337] req@ffff995b83506a00 x1855007616364544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075337.705114:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506a00 x1855007616364544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075337.705122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075337.705125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075337.705128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616364544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075337.705135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075334/real 1769075337] req@ffff995b83505c00 x1855007616364416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075337.705145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505c00 x1855007616364416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075337.705152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075337.705156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075337.705159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616364416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075337.705174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075334/real 1769075337] req@ffff995b83505180 x1855007616364160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075337.705219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616364160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075337.705228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075337.705232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075337.705236:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616364160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075337.705244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075334/real 1769075337] req@ffff995b83504e00 x1855007616364288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075337.705259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616364288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075337.705268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075337.705272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075337.705275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616364288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075337.705282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075334/real 1769075337] req@ffff995b83507800 x1855007616364032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075337.705294:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616364032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075337.705301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075337.705318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075337.705321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616364032:192.168.204.132@tcp:250:kworker.0 00010000:00010000:3.0:1769075338.408561:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 13/13 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1769075338.408648:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075338.408977:0:2418:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83505500 x1855007616364928/t0(0) o103->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 416/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075338.728563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075338.728629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075339.752529:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075339.752600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075339.752601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075339.752656:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616365056:192.168.204.132@tcp:250:kworker.0 00000800:00000100:3.0:1769075339.752727:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075339.752765:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075339/real 1769075339] req@ffff995b83506300 x1855007616365056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075339.752784:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616365056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075339.752802:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075339.752806:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075339.752811:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616365056:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075339.752839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616365184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075339.752863:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075339.752876:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075339/real 1769075339] req@ffff995b83507100 x1855007616365184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075339.752889:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616365184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075339.752898:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075339.752902:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075339.752905:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616365184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075339.752912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616365312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075339.752932:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075339.752944:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075339/real 1769075339] req@ffff995b83505880 x1855007616365312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075339.752958:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505880 x1855007616365312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075339.752974:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075339.752978:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075339.752982:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616365312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075339.752989:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616365440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075339.753012:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075339.753025:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075339/real 1769075339] req@ffff995b83504000 x1855007616365440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075339.753039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616365440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075339.753047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075339.753051:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075339.753055:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616365440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075339.753061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616365568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075339.753085:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075339.753097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075339/real 1769075339] req@ffff995b83507b80 x1855007616365568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075339.753111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507b80 x1855007616365568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075339.753118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075339.753121:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075339.753124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616365568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075339.753147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616365696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075339.753213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075339.753225:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075339/real 1769075339] req@ffff995b83504a80 x1855007616365696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075339.753238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504a80 x1855007616365696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075339.753247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075339.753250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075339.753253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616365696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075339.753258:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616365824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075339.753278:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075339.753289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075339/real 1769075339] req@ffff995b83505f80 x1855007616365824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075339.753301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505f80 x1855007616365824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075339.753310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075339.753317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075339.753321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616365824:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075340.456590:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075340.776586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075340.776629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075340.776654:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075341.800548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075341.800550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075342.824558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075342.824584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075343.528628:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075343.848549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075343.848552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075344.872570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075344.872603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075345.896553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075345.896578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075346.600590:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075346.920535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075346.920627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075347.944528:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075347.944531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075348.968510:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075348.968538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075349.672621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075349.992629:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075349.992637:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075349.992642:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075349.992653:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2335 00000100:00080000:1.0:1769075349.992664:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075349.992671:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075349.992704:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5880 x1855007616366848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075349.992734:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075349.992740:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075349.992743:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075349.992749:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2335 00000100:00080000:1.0:1769075349.992756:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075349.992762:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075349.992774:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075349.992797:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616366976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075349.992807:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075349.992809:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075349.992811:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075349.992815:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2335 00000100:00080000:1.0:1769075349.992818:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075349.992821:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075349.992827:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075349.992838:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616367104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075349.992844:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075349.992847:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075349.992848:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075349.992851:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2335 00000100:00080000:1.0:1769075349.992853:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075349.992856:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075349.992861:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075349.992872:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616367232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075349.992882:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075349.992890:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075349.992893:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075349.992900:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2335 00000100:00080000:1.0:1769075349.992908:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075349.992914:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075349.992925:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075349.992935:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616367360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075349.992947:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075349.992950:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075349.992952:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075349.992955:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2335 00000100:00080000:1.0:1769075349.992959:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075349.992962:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075349.992969:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075349.992992:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7b80 x1855007616367488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075349.993017:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075349.993029:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075349.993033:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075349.993042:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2335 00000100:00080000:1.0:1769075349.993052:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075349.993068:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075349.993074:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075349.993091:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6680 x1855007616367616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075351.016535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075351.016549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075352.040541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075352.040541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075352.040598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075352.744658:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075353.064581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075353.064590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075353.064656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075354.088539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075354.088539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075354.088770:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075355.112560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075355.112567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075355.112602:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075355.112609:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075355.112614:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075355.112624:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2340 00000100:00080000:3.0:1769075355.112635:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075355.112643:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075355.112677:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616367744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075355.112704:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075355.112709:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075355.112711:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075355.112715:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2340 00000100:00080000:3.0:1769075355.112720:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075355.112724:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075355.112734:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075355.112749:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616367872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075355.112763:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075355.112772:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075355.112777:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075355.112782:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2340 00000100:00080000:3.0:1769075355.112787:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075355.112792:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075355.112799:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075355.112812:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616368000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075355.112824:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075355.112827:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075355.112829:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075355.112838:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2340 00000100:00080000:3.0:1769075355.112844:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075355.112850:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075355.112856:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075355.112869:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616368128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075355.112913:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075355.112923:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075355.112928:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075355.112938:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2340 00000100:00080000:3.0:1769075355.112947:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075355.112956:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075355.112965:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075355.112980:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616368256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075355.112993:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075355.113003:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075355.113005:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075355.113011:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2340 00000100:00080000:3.0:1769075355.113016:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075355.113021:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075355.113027:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075355.113045:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616368384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075355.113062:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075355.113070:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075355.113074:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075355.113082:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2340 00000100:00080000:3.0:1769075355.113088:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075355.113098:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075355.113107:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075355.113124:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616368512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075355.816645:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075356.136568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075356.136577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075357.160564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075357.160564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075358.184531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075358.184531:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075358.184583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075358.888693:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075359.208562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075359.208670:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075360.232552:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075360.232564:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075360.232570:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075360.232582:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2345 00000100:00100000:3.0:1769075360.232600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075360.232603:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075360.232621:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075360.232665:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5180 x1855007616368640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075360.232692:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075360.232696:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075360.232698:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075360.232703:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2345 00000100:00080000:1.0:1769075360.232708:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075360.232712:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075360.232720:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075360.232736:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6680 x1855007616368768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075360.232757:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075360.232759:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075360.232761:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075360.232765:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2345 00000100:00080000:1.0:1769075360.232768:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075360.232770:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075360.232774:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075360.232786:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7b80 x1855007616368896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075360.232793:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075360.232795:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075360.232797:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075360.232799:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2345 00000100:00080000:1.0:1769075360.232801:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075360.232804:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075360.232828:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075360.232837:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616369024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075360.232847:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075360.232850:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075360.232851:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075360.232853:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2345 00000100:00080000:1.0:1769075360.232856:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075360.232863:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075360.232873:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075360.232880:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616369152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075360.232885:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075360.232888:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075360.232890:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075360.232893:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2345 00000100:00080000:1.0:1769075360.232898:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075360.232901:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075360.232905:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075360.232912:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616369280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075360.232921:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075360.232924:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075360.232925:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075360.232930:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2345 00000100:00080000:1.0:1769075360.232938:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075360.232952:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075360.232963:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075360.232975:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616369408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075361.256574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075361.256615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075361.256633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075361.960612:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075362.280574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075362.280574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075363.304526:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075363.304526:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075363.304594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075363.304594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075364.328577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075364.328585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075364.328613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075365.032634:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075365.352574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075365.352575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075365.352794:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616369536:192.168.204.132@tcp:250:kworker.0 00000800:00000100:3.0:1769075365.352885:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075365.352947:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075365/real 1769075365] req@ffff995b85ac9f80 x1855007616369536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075365.352995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9f80 x1855007616369536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075365.353049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075365.353059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075365.353065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616369536:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075365.353092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616369664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075365.353121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075365.353137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075365/real 1769075365] req@ffff995b85ac8a80 x1855007616369664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075365.353152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616369664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075365.353162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075365.353167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075365.353171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616369664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075365.353178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616369792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075365.353199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075365.353212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075365/real 1769075365] req@ffff995b85acb800 x1855007616369792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075365.353225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616369792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075365.353235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075365.353239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075365.353243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616369792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075365.353250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616369920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075365.353270:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075365.353281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075365/real 1769075365] req@ffff995b85acad80 x1855007616369920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075365.353293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acad80 x1855007616369920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075365.353301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075365.353305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075365.353308:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616369920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075365.353315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616370048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075365.353335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075365.353346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075365/real 1769075365] req@ffff995b85ac9500 x1855007616370048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075365.353359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9500 x1855007616370048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075365.353367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075365.353371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075365.353374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616370048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075365.353381:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616370176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075365.353400:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075365.353412:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075365/real 1769075365] req@ffff995b85ac9880 x1855007616370176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075365.353425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616370176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075365.353435:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075365.353439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075365.353442:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616370176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075365.353449:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616370304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075365.353501:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075365.353519:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075365/real 1769075365] req@ffff995b85acbb80 x1855007616370304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075365.353543:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616370304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075365.353572:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075365.353577:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075365.353587:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616370304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075366.376567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075366.376609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075367.400513:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075367.400532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075367.400581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075367.400585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075368.104574:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075368.424613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075368.424623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075369.448573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075369.448573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075370.472584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075370.472584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075370.472667:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075370.472672:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075370.472676:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075370.472684:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2356 00000100:00080000:3.0:1769075370.472694:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075370.472700:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075370.472744:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616370432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075370.472769:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075370.472772:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075370.472774:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075370.472777:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2356 00000100:00080000:3.0:1769075370.472779:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075370.472782:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075370.472789:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075370.472809:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616370560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075370.472816:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075370.472819:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075370.472821:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075370.472823:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2356 00000100:00080000:3.0:1769075370.472825:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075370.472828:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075370.472831:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075370.472844:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616370688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075370.472866:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075370.472868:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075370.472870:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075370.472874:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2356 00000100:00080000:3.0:1769075370.472877:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075370.472880:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075370.472895:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075370.472909:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616370816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075370.472915:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075370.472918:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075370.472919:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075370.472922:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2356 00000100:00080000:3.0:1769075370.472927:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075370.472929:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075370.472931:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075370.472938:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616370944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075370.472944:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075370.472946:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075370.472947:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075370.472961:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2356 00000100:00080000:3.0:1769075370.472975:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075370.472979:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075370.472982:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075370.472989:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616371072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075370.472995:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075370.473004:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075370.473014:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075370.473019:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2356 00000100:00080000:3.0:1769075370.473021:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075370.473024:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075370.473027:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075370.473043:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616371200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075371.176632:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075371.496602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075371.496602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075372.520519:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075372.520637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075373.544550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075373.544550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075374.248648:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075374.568581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075374.568581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075374.568672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075375.592582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075375.592583:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075375.592594:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075375.592599:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075375.592613:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2361 00000100:00080000:3.0:1769075375.592627:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075375.592636:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075375.592675:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075375.592678:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616371328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075375.592698:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075375.592701:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075375.592703:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075375.592707:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2361 00000100:00080000:3.0:1769075375.592710:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075375.592714:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075375.592723:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075375.592734:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616371456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075375.592779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075375.592783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075375.592786:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075375.592792:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2361 00000100:00080000:3.0:1769075375.592795:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075375.592799:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075375.592804:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075375.592833:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616371584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075375.592853:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075375.592868:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075375.592873:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075375.592881:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2361 00000100:00080000:3.0:1769075375.592889:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075375.592912:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075375.592922:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075375.592960:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616371712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075375.593000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075375.593004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075375.593007:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075375.593013:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2361 00000100:00080000:3.0:1769075375.593022:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075375.593029:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075375.593038:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075375.593053:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616371840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075375.593063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075375.593066:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075375.593067:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075375.593072:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2361 00000100:00080000:3.0:1769075375.593075:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075375.593083:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075375.593096:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075375.593108:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616371968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075375.593171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075375.593201:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075375.593209:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075375.593222:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2361 00000100:00080000:3.0:1769075375.593233:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075375.593243:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075375.593275:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075375.593304:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b800 x1855007616372096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075376.616568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075376.616580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075376.616609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075377.320705:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075377.640593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075377.640593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075378.664639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075378.664652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075379.688619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075379.688621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075380.392634:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075380.712587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075380.712587:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075380.712601:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075380.712608:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075380.712624:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2366 00000100:00080000:3.0:1769075380.712638:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075380.712642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075380.712649:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075380.712690:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558380 x1855007616372224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075380.712711:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075380.712715:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075380.712717:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075380.712722:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2366 00000100:00080000:3.0:1769075380.712727:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075380.712731:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075380.712740:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075380.712769:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559180 x1855007616372352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075380.712778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075380.712783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075380.712786:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075380.712795:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2366 00000100:00080000:3.0:1769075380.712799:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075380.712802:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075380.712831:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075380.712841:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b800 x1855007616372480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075380.712847:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075380.712849:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075380.712851:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075380.712853:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2366 00000100:00080000:3.0:1769075380.712856:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075380.712859:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075380.712862:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075380.712870:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616372608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075380.712897:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075380.712901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075380.712903:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075380.712906:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2366 00000100:00080000:3.0:1769075380.712911:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075380.712918:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075380.712926:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075380.712939:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616372736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075380.712947:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075380.712949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075380.712951:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075380.712954:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2366 00000100:00080000:3.0:1769075380.712957:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075380.712961:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075380.712965:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075380.712993:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616372864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075380.713144:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075380.713152:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075380.713156:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075380.713166:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2366 00000100:00080000:3.0:1769075380.713301:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075380.713312:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075380.713346:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075380.713508:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616372992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075381.736637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075381.736739:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075382.760607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075382.760624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075383.464590:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075383.784571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075383.784571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075384.808515:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075384.808543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075385.832545:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075385.832546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075385.832556:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075385.832560:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075385.832569:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2371 00000100:00080000:3.0:1769075385.832579:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075385.832584:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075385.832604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075385.832618:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616373120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075385.832639:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075385.832642:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075385.832643:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075385.832647:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2371 00000100:00080000:3.0:1769075385.832649:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075385.832653:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075385.832661:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075385.832673:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616373248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075385.832682:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075385.832685:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075385.832686:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075385.832689:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2371 00000100:00080000:3.0:1769075385.832691:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075385.832693:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075385.832696:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075385.832705:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616373376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075385.832712:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075385.832715:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075385.832717:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075385.832719:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2371 00000100:00080000:3.0:1769075385.832721:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075385.832724:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075385.832731:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075385.832742:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616373504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075385.832748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075385.832751:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075385.832753:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075385.832755:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2371 00000100:00080000:3.0:1769075385.832757:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075385.832759:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075385.832762:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075385.832770:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616373632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075385.832779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075385.832785:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075385.832788:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075385.832796:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2371 00000100:00080000:3.0:1769075385.832801:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075385.832805:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075385.832808:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075385.832814:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616373760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075385.832821:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075385.832823:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075385.832825:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075385.832830:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2371 00000100:00080000:3.0:1769075385.832832:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075385.832836:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075385.832842:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075385.832851:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616373888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075386.536609:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075386.856573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075386.856589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075386.856611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075387.880583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075388.904545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075388.904574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075389.608563:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075389.928595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075389.928614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075389.928776:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075390.952584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075390.952586:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075390.952598:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075390.952603:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075390.952613:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2376 00000100:00080000:3.0:1769075390.952625:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075390.952633:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075390.952671:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616374016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075390.952707:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075390.952712:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075390.952715:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075390.952720:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2376 00000100:00080000:3.0:1769075390.952725:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075390.952731:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075390.952742:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075390.952758:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616374144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075390.952771:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075390.952775:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075390.952777:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075390.952781:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2376 00000100:00080000:3.0:1769075390.952784:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075390.952788:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075390.952794:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075390.952806:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616374272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075390.952816:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075390.952830:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075390.952833:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075390.952837:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2376 00000100:00080000:3.0:1769075390.952840:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075390.952844:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075390.952851:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075390.952862:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616374400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075390.952872:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075390.952876:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075390.952878:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075390.952882:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2376 00000100:00080000:3.0:1769075390.952886:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075390.952890:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075390.952896:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075390.952906:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616374528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075390.952917:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075390.952921:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075390.952924:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075390.952929:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2376 00000100:00080000:3.0:1769075390.952933:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075390.952937:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075390.952942:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075390.952953:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616374656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075390.952964:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075390.952968:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075390.952970:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075390.952975:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2376 00000100:00080000:3.0:1769075390.952979:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075390.952983:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075390.952989:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075390.953001:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616374784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075391.976556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075391.976582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075392.680613:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075393.000558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075393.000595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075393.000611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075394.024595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075394.024637:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075394.024702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075395.048676:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075395.048788:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075395.752774:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075396.072565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075396.072585:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769075396.072594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075396.072601:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075396.072611:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075396.072629:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2381 00000100:00080000:1.0:1769075396.072648:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075396.072661:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075396.072727:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5880 x1855007616374912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075396.072786:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075396.072792:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075396.072796:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075396.072803:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2381 00000100:00080000:1.0:1769075396.072809:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075396.072820:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075396.072848:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075396.072872:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616375040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075396.072888:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075396.072893:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075396.072896:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075396.072900:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2381 00000100:00080000:1.0:1769075396.072908:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075396.072919:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075396.072933:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075396.072951:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616375168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075396.072969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075396.072974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075396.072981:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075396.072999:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2381 00000100:00080000:1.0:1769075396.073045:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075396.073060:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075396.073079:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075396.073126:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616375296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075396.073154:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075396.073170:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075396.073174:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075396.073184:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2381 00000100:00080000:1.0:1769075396.073192:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075396.073252:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075396.073266:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075396.073295:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616375424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075396.073309:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075396.073323:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075396.073345:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075396.073357:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2381 00000100:00080000:1.0:1769075396.073369:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075396.073379:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075396.073390:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075396.073415:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616375552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075396.073458:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075396.073464:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075396.073467:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075396.073548:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2381 00000100:00080000:1.0:1769075396.073561:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075396.073570:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075396.073577:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075396.073600:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616375680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075397.096553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075397.096601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075398.120549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075398.120550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075398.120614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075398.824643:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075399.144542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075399.144549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075400.168523:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075400.168560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075401.192562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075401.192563:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075401.192575:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075401.192581:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075401.192592:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2386 00000100:00080000:1.0:1769075401.192615:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075401.192624:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075401.192662:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616375808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075401.192692:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075401.192696:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075401.192699:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075401.192704:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2386 00000100:00080000:1.0:1769075401.192709:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075401.192714:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075401.192724:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075401.192744:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616375936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075401.192756:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075401.192760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075401.192762:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075401.192766:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2386 00000100:00080000:1.0:1769075401.192770:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075401.192775:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075401.192779:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075401.192790:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616376064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075401.192799:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075401.192803:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075401.192805:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075401.192809:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2386 00000100:00080000:1.0:1769075401.192813:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075401.192818:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075401.192825:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075401.192838:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616376192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075401.192858:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075401.192862:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075401.192864:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075401.192870:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2386 00000100:00080000:1.0:1769075401.192875:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075401.192884:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075401.192891:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075401.192907:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616376320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075401.192924:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075401.192928:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075401.192931:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075401.192937:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2386 00000100:00080000:1.0:1769075401.192942:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075401.192950:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075401.192957:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075401.192972:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616376448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075401.192990:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075401.192995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075401.192997:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075401.193002:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2386 00000100:00080000:1.0:1769075401.193009:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075401.193017:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075401.193023:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075401.193037:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616376576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075401.896719:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075402.216557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075402.216590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075403.240604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075403.240604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075403.240665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075404.264556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075404.264592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:1.0:1769075404.264826:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769075404.264881:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769075404.264908:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075404.264964:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075404.264971:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075404.264976:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075404.264981:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075404.264988:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075404.264996:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00010000:00010000:1.0:1769075404.968677:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075405.288596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075405.288623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075406.312552:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075406.312560:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075406.312565:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075406.312581:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2392 00000100:00100000:3.0:1769075406.312582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075406.312595:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075406.312606:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075406.312644:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5880 x1855007616376704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075406.312687:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075406.312694:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075406.312696:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075406.312703:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2392 00000100:00080000:1.0:1769075406.312707:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075406.312712:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075406.312723:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075406.312750:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616376832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075406.312763:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075406.312767:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075406.312769:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075406.312774:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2392 00000100:00080000:1.0:1769075406.312779:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075406.312782:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075406.312787:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075406.312801:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616376960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075406.312816:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075406.312821:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075406.312823:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075406.312828:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2392 00000100:00080000:1.0:1769075406.312832:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075406.312837:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075406.312844:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075406.312859:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616377088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075406.312883:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075406.312887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075406.312889:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075406.312896:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2392 00000100:00080000:1.0:1769075406.312903:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075406.312921:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075406.312930:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075406.312946:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616377216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075406.312958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075406.312962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075406.312965:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075406.312973:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2392 00000100:00080000:1.0:1769075406.312988:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075406.312992:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075406.312998:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075406.313020:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7b80 x1855007616377344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075406.313055:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075406.313060:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075406.313062:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075406.313067:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2392 00000100:00080000:1.0:1769075406.313071:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075406.313080:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075406.313090:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075406.313105:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6680 x1855007616377472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075407.336605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075407.336638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075407.336649:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075408.040604:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075408.360562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075408.360563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075409.384543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075409.384584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075410.408572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075410.408600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075411.112645:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075411.432563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075412.456531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075412.456583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075413.480552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075414.184597:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075414.504549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075414.504550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075415.528516:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075415.528529:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075416.552539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075416.552559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075416.552609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075416.552627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075417.256735:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075417.576514:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075417.576566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075417.576594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075418.600527:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075418.600528:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075419.624541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075419.624549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075420.328628:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075420.648545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075420.648545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075421.672535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075421.672549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075421.672571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075422.696601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075422.696628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075423.400606:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075423.720604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075423.720606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075424.744540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075424.744578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075425.768834:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075426.472571:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075426.792555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075426.792580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075426.792604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075427.816566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075427.816581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075427.816612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075428.840539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075428.840567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075428.840625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075428.840717:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075429.544711:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075429.864598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075430.888605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075430.888691:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075431.912542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075431.912544:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075431.912597:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075431.912635:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075431.912640:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075431.912649:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2417 00000100:00080000:1.0:1769075431.912659:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075431.912665:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075431.912695:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5180 x1855007616381184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075431.912717:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075431.912720:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075431.912721:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075431.912724:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2417 00000100:00080000:1.0:1769075431.912727:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075431.912730:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075431.912738:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075431.912752:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6680 x1855007616381312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075431.912762:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075431.912764:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075431.912765:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075431.912768:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2417 00000100:00080000:1.0:1769075431.912770:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075431.912772:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075431.912775:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075431.912784:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7b80 x1855007616381440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075431.912790:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075431.912792:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075431.912794:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075431.912796:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2417 00000100:00080000:1.0:1769075431.912804:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075431.912807:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075431.912810:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075431.912821:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616381568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075431.912833:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075431.912835:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075431.912837:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075431.912840:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2417 00000100:00080000:1.0:1769075431.912843:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075431.912846:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075431.912851:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075431.912862:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616381696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075431.912869:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075431.912872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075431.912876:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075431.912884:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2417 00000100:00080000:1.0:1769075431.912887:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075431.912892:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075431.912897:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075431.912906:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616381824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075431.912913:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075431.912915:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075431.912917:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075431.912919:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2417 00000100:00080000:1.0:1769075431.912922:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075431.912924:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075431.912927:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075431.912937:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616381952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075432.616622:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075432.936668:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075432.936697:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075433.960631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075434.984608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075434.984667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075435.688673:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075436.008668:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075436.008723:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075437.032579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075437.032582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075437.032644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075438.056583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075438.056704:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075438.760630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075439.080591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075439.080600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075440.104621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075440.104622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075441.128618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075441.128636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075441.832732:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075442.152635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075442.152682:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075442.152734:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075442.152751:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075442.152761:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075442.152778:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2427 00000100:00080000:3.0:1769075442.152792:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075442.152802:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075442.152850:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616382976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075442.152889:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075442.152894:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075442.152897:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075442.152902:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2427 00000100:00080000:3.0:1769075442.152906:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075442.152912:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075442.152924:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075442.152941:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616383104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075442.152962:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075442.152975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075442.152978:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075442.152983:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2427 00000100:00080000:3.0:1769075442.152987:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075442.152992:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075442.152998:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075442.153013:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616383232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075442.153026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075442.153031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075442.153034:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075442.153038:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2427 00000100:00080000:3.0:1769075442.153042:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075442.153046:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075442.153051:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075442.153067:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616383360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075442.153090:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075442.153096:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075442.153100:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075442.153107:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2427 00000100:00080000:3.0:1769075442.153125:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075442.153130:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075442.153135:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075442.153149:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616383488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075442.153163:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075442.153171:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075442.153178:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075442.153193:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2427 00000100:00080000:3.0:1769075442.153203:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075442.153209:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075442.153216:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075442.153263:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616383616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075442.153296:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075442.153301:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075442.153304:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075442.153309:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2427 00000100:00080000:3.0:1769075442.153314:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075442.153323:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075442.153330:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075442.153341:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b800 x1855007616383744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075443.176540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075443.176559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075443.176618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075444.200595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075444.200598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075444.200653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075444.200693:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075444.904694:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075445.224542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075445.224598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075446.248575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075446.248576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075447.272821:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075447.272830:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075447.272835:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075447.272845:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2432 00000100:00080000:3.0:1769075447.272874:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075447.272882:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075447.272915:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559180 x1855007616383872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075447.272956:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075447.272959:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075447.272961:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075447.272966:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2432 00000100:00080000:3.0:1769075447.272970:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075447.272976:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075447.272986:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075447.273038:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558380 x1855007616384000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075447.273065:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075447.273069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075447.273072:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075447.273080:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2432 00000100:00080000:3.0:1769075447.273086:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075447.273092:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075447.273100:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075447.273116:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b800 x1855007616384128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075447.273126:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075447.273129:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075447.273130:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075447.273133:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2432 00000100:00080000:3.0:1769075447.273136:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075447.273139:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075447.273143:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075447.273154:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616384256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075447.273163:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075447.273166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075447.273167:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075447.273171:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2432 00000100:00080000:3.0:1769075447.273174:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075447.273177:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075447.273182:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075447.273195:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616384384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075447.273203:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075447.273208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075447.273210:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075447.273248:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2432 00000100:00080000:3.0:1769075447.273253:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075447.273266:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075447.273281:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075447.273310:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616384512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075447.273334:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075447.273338:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075447.273341:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075447.273400:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2432 00000100:00080000:3.0:1769075447.273408:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075447.273420:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075447.273429:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075447.273457:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616384640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075447.273611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075447.976625:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075448.296603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075448.296620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075449.320534:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075449.320567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075450.344537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075450.344567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075450.344599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075451.048644:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075451.368589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075451.368589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075452.392563:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075452.392565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075452.392574:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075452.392580:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075452.392590:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2438 00000100:00080000:3.0:1769075452.392602:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075452.392620:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075452.392657:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616384768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075452.392693:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075452.392697:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075452.392698:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075452.392702:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2438 00000100:00080000:3.0:1769075452.392705:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075452.392708:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075452.392717:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075452.392729:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616384896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075452.392762:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075452.392766:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075452.392769:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075452.392776:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2438 00000100:00080000:3.0:1769075452.392781:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075452.392786:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075452.392794:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075452.392834:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616385024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075452.392862:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075452.392866:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075452.392868:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075452.392873:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2438 00000100:00080000:3.0:1769075452.392879:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075452.392885:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075452.392896:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075452.392917:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616385152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075452.392936:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075452.392943:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075452.392948:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075452.392964:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2438 00000100:00080000:3.0:1769075452.392969:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075452.392972:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075452.392978:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075452.392991:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616385280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075452.393007:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075452.393012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075452.393015:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075452.393022:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2438 00000100:00080000:3.0:1769075452.393031:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075452.393039:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075452.393044:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075452.393058:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616385408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075452.393101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075452.393111:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075452.393117:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075452.393131:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2438 00000100:00080000:3.0:1769075452.393139:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075452.393147:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075452.393152:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075452.393169:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616385536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075453.416568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075453.416630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075454.120586:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075454.440560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075454.440591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075454.440641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075455.464589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075455.464676:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075456.488609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075456.488618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075457.192770:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075457.512646:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769075457.512667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075457.512668:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075457.512680:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075457.512697:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2443 00000100:00080000:1.0:1769075457.512714:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769075457.512718:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075457.512727:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075457.512784:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5880 x1855007616385664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075457.512832:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075457.512840:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075457.512844:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075457.512852:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2443 00000100:00080000:1.0:1769075457.512861:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075457.512870:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075457.512887:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075457.512943:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616385792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075457.512970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075457.512976:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075457.512979:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075457.512985:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2443 00000100:00080000:1.0:1769075457.512992:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075457.513007:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075457.513014:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075457.513031:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616385920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075457.513047:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075457.513059:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075457.513063:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075457.513073:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2443 00000100:00080000:1.0:1769075457.513095:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075457.513104:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075457.513116:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075457.513145:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616386048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075457.513211:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075457.513219:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075457.513224:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075457.513240:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2443 00000100:00080000:1.0:1769075457.513255:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075457.513262:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075457.513288:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075457.513314:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616386176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075457.513332:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075457.513337:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075457.513339:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075457.513345:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2443 00000100:00080000:1.0:1769075457.513349:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075457.513360:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075457.513367:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075457.513391:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616386304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075457.513406:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075457.513412:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075457.513415:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075457.513429:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2443 00000100:00080000:1.0:1769075457.513438:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075457.513444:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075457.513451:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075457.513504:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616386432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075458.216655:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 16/16 locks to regular blp list, 1 blwis in pool 00000100:00080000:1.0:1769075458.220018:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83506d80 x1855007616386560/t0(0) o103->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 440/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075458.536634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075458.536637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075458.536686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075459.240732:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075459.560574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075459.560714:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075460.584561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075460.584632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075460.584638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075461.608619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075461.608622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075462.312673:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075462.632567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075462.632642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075462.632664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075463.656612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075464.680686:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075464.680746:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075465.384612:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075465.704595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075465.704617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075465.704666:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075466.728586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075467.752573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075467.752632:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075467.752637:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075467.752640:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075467.752656:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2453 00000100:00080000:3.0:1769075467.752667:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075467.752674:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075467.752702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075467.752718:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616387584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075467.752739:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075467.752743:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075467.752744:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075467.752750:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2453 00000100:00080000:3.0:1769075467.752753:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075467.752757:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075467.752764:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075467.752779:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616387712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075467.752793:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075467.752795:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075467.752796:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075467.752799:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2453 00000100:00080000:3.0:1769075467.752802:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075467.752804:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075467.752807:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075467.752817:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616387840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075467.752823:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075467.752826:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075467.752827:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075467.752829:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2453 00000100:00080000:3.0:1769075467.752831:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075467.752834:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075467.752837:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075467.752844:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616387968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075467.752851:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075467.752853:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075467.752855:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075467.752857:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2453 00000100:00080000:3.0:1769075467.752860:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075467.752864:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075467.752868:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075467.752877:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616388096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075467.752886:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075467.752892:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075467.752893:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075467.752897:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2453 00000100:00080000:3.0:1769075467.752900:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075467.752903:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075467.752907:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075467.752929:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616388224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075467.752937:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075467.752940:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075467.752942:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075467.752949:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2453 00000100:00080000:3.0:1769075467.752953:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075467.752956:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075467.752961:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075467.752974:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616388352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075468.456657:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075468.776563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075468.776577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075468.776614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075469.800574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075469.800574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075469.800649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075470.824612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075470.824612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075470.824678:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075471.528755:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075471.848567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075471.848567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075472.872568:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075472.872583:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769075472.872584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075472.872589:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075472.872603:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2458 00000100:00080000:3.0:1769075472.872617:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075472.872627:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075472.872642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075472.872672:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616388480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075472.872714:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075472.872719:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075472.872722:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075472.872727:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2458 00000100:00080000:3.0:1769075472.872733:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075472.872738:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075472.872749:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075472.872765:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616388608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075472.872776:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075472.872780:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075472.872782:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075472.872786:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2458 00000100:00080000:3.0:1769075472.872790:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075472.872795:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075472.872800:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075472.872810:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616388736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075472.872822:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075472.872826:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075472.872829:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075472.872834:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2458 00000100:00080000:3.0:1769075472.872838:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075472.872846:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075472.872853:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075472.872870:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616388864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075472.872881:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075472.872885:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075472.872888:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075472.872892:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2458 00000100:00080000:3.0:1769075472.872896:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075472.872900:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075472.872907:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075472.872956:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616388992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075472.872969:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075472.872974:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075472.872981:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075472.872993:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2458 00000100:00080000:3.0:1769075472.873000:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075472.873007:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075472.873015:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075472.873031:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616389120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075472.873045:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075472.873056:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075472.873060:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075472.873070:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2458 00000100:00080000:3.0:1769075472.873077:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075472.873085:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075472.873096:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075472.873116:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b800 x1855007616389248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075473.896614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075473.896653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075474.600680:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075474.920567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075474.920570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075474.920667:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075475.944602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075475.944621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075475.944657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:3.0:1769075476.648651:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1769075476.648785:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075476.968590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075476.968667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075477.992615:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769075477.992620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075477.992624:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075477.992632:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075477.992646:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2463 00000100:00100000:3.0:1769075477.992656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075477.992660:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075477.992671:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075477.992723:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616389504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075477.992749:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075477.992754:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075477.992756:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075477.992761:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2463 00000100:00080000:1.0:1769075477.992765:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075477.992770:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075477.992781:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075477.992798:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616389632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075477.992807:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075477.992811:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075477.992813:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075477.992816:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2463 00000100:00080000:1.0:1769075477.992819:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075477.992823:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075477.992827:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075477.992866:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616389760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075477.992881:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075477.992884:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075477.992886:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075477.992890:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2463 00000100:00080000:1.0:1769075477.992894:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075477.992897:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075477.992902:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075477.992912:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616389888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075477.992921:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075477.992924:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075477.992925:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075477.992929:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2463 00000100:00080000:1.0:1769075477.992932:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075477.992936:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075477.992940:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075477.992949:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616390016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075477.992958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075477.992962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075477.992964:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075477.992968:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2463 00000100:00080000:1.0:1769075477.992972:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075477.992976:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075477.992981:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075477.992990:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616390144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075477.992998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075477.993002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075477.993004:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075477.993008:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2463 00000100:00080000:1.0:1769075477.993011:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075477.993015:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075477.993020:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075477.993030:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616390272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075478.696691:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 36/36 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1769075478.696742:0:13603:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 36 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1769075478.696749:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 16/16 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1769075478.696755:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b80030a00/0xe83a7ce271e22eb6 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdd1a6 expref: -99 pid: 65488 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.696850:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x62:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.696924:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f79c00/0xe83a7ce271e22e77 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdd0f7 expref: -99 pid: 65483 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.696956:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x61:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.696977:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86218c00/0xe83a7ce271e22e38 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdd048 expref: -99 pid: 65478 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697021:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x60:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697053:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f79a00/0xe83a7ce271e22df2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdcf84 expref: -99 pid: 65472 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697070:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x5f:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697119:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486c400/0xe83a7ce271e22db3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdced5 expref: -99 pid: 65467 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697128:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x5e:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697140:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f79800/0xe83a7ce271e22d74 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdce26 expref: -99 pid: 65462 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697149:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x5d:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697195:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229200/0xe83a7ce271e22cfd lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdcd62 expref: -99 pid: 65453 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697211:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x5c:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697229:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b80030000/0xe83a7ce271e22c1d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdcc89 expref: -99 pid: 65443 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697269:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x5b:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697324:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228c00/0xe83a7ce271e22bde lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdcbda expref: -99 pid: 65438 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697341:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x5a:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697378:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f51800/0xe83a7ce271e22b9f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdcb2b expref: -99 pid: 65433 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697387:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x59:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697400:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228800/0xe83a7ce271e22b59 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdca67 expref: -99 pid: 65428 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697431:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x58:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697457:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229a00/0xe83a7ce271e22b1a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc9b8 expref: -99 pid: 65422 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697470:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x57:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697589:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86218000/0xe83a7ce271e22aa3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc8f4 expref: -99 pid: 65413 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697621:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x56:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697663:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229800/0xe83a7ce271e229c3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc81b expref: -99 pid: 65402 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697685:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x55:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697732:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486ce00/0xe83a7ce271e2291b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x54:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc757 expref: -99 pid: 65395 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697750:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x54:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697763:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229000/0xe83a7ce271e228dc lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc6a8 expref: -99 pid: 65389 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697810:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x53:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697867:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229400/0xe83a7ce271e2289d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc5f9 expref: -99 pid: 65384 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697888:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x52:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697911:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86219a00/0xe83a7ce271e2285e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc54a expref: -99 pid: 65379 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.697930:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x51:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.697967:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228600/0xe83a7ce271e227e7 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc486 expref: -99 pid: 65370 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698028:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x50:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698075:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228e00/0xe83a7ce271e22707 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc3ad expref: -99 pid: 65358 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698125:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x4f:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698137:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486c000/0xe83a7ce271e2265f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc2e9 expref: -99 pid: 65351 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698162:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x4e:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698197:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229e00/0xe83a7ce271e22620 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc23a expref: -99 pid: 65346 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698217:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x4d:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698247:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375400/0xe83a7ce271e225e1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc18b expref: -99 pid: 65341 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698309:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x4c:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698366:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375200/0xe83a7ce271e225a2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc0dc expref: -99 pid: 65335 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698379:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x4b:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698390:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83374200/0xe83a7ce271e22563 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdc02d expref: -99 pid: 65330 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698400:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x4a:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698453:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486c600/0xe83a7ce271e22524 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdbf7e expref: -99 pid: 65324 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698487:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x49:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698506:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375600/0xe83a7ce271e224de lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdbeba expref: -99 pid: 65319 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698572:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x48:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698615:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375c00/0xe83a7ce271e2249f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdbe0b expref: -99 pid: 65313 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698688:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x47:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698704:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83374c00/0xe83a7ce271e22428 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdbd47 expref: -99 pid: 65304 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698720:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x46:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698731:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ff600/0xe83a7ce271e22348 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdbc6e expref: -99 pid: 65293 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698797:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x45:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698838:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375000/0xe83a7ce271e22268 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdbb95 expref: -99 pid: 65282 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698885:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x44:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698901:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ff000/0xe83a7ce271e22150 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdbaa7 expref: -99 pid: 65270 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.698933:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x43:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.698979:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b72200/0xe83a7ce271e22118 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdba0d expref: -99 pid: 65265 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.699042:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x42:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.699108:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b73600/0xe83a7ce271e220e0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdb96c expref: -99 pid: 65260 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.699147:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x41:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.699201:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ffe00/0xe83a7ce271e220a8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdb8d2 expref: -99 pid: 65255 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075478.699217:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x40:0x0]: losing layout lock 00010000:00010000:1.0:1769075478.699237:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b813c8400/0xe83a7ce271e22085 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x6:0x0].0x0 bits 0x10/0x0 rrc: 7 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x557988ac98fdb862 expref: -99 pid: 65254 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1769075478.699291:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b80030a00/0xe83a7ce271e22eb6 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdd1a6 expref: -99 pid: 65488 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699312:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f79c00/0xe83a7ce271e22e77 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdd0f7 expref: -99 pid: 65483 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699330:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86218c00/0xe83a7ce271e22e38 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdd048 expref: -99 pid: 65478 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699371:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f79a00/0xe83a7ce271e22df2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcf84 expref: -99 pid: 65472 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699397:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486c400/0xe83a7ce271e22db3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdced5 expref: -99 pid: 65467 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699403:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f79800/0xe83a7ce271e22d74 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdce26 expref: -99 pid: 65462 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699409:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229200/0xe83a7ce271e22cfd lrc: 1/0,0 mode: --/PR res: [0x240000403:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcd62 expref: -99 pid: 65453 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699473:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b80030000/0xe83a7ce271e22c1d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcc89 expref: -99 pid: 65443 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699484:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228c00/0xe83a7ce271e22bde lrc: 1/0,0 mode: --/PR res: [0x240000403:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcbda expref: -99 pid: 65438 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699501:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f51800/0xe83a7ce271e22b9f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcb2b expref: -99 pid: 65433 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699531:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228800/0xe83a7ce271e22b59 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdca67 expref: -99 pid: 65428 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699561:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229a00/0xe83a7ce271e22b1a lrc: 1/0,0 mode: --/PR res: [0x240000403:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc9b8 expref: -99 pid: 65422 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699581:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86218000/0xe83a7ce271e22aa3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc8f4 expref: -99 pid: 65413 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699599:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229800/0xe83a7ce271e229c3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc81b expref: -99 pid: 65402 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699603:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486ce00/0xe83a7ce271e2291b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x54:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc757 expref: -99 pid: 65395 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699608:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229000/0xe83a7ce271e228dc lrc: 1/0,0 mode: --/PR res: [0x240000403:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc6a8 expref: -99 pid: 65389 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699613:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229400/0xe83a7ce271e2289d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc5f9 expref: -99 pid: 65384 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699621:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86219a00/0xe83a7ce271e2285e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc54a expref: -99 pid: 65379 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699631:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228600/0xe83a7ce271e227e7 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc486 expref: -99 pid: 65370 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699640:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228e00/0xe83a7ce271e22707 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc3ad expref: -99 pid: 65358 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699647:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486c000/0xe83a7ce271e2265f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc2e9 expref: -99 pid: 65351 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699652:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229e00/0xe83a7ce271e22620 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc23a expref: -99 pid: 65346 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699657:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375400/0xe83a7ce271e225e1 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc18b expref: -99 pid: 65341 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699661:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375200/0xe83a7ce271e225a2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc0dc expref: -99 pid: 65335 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699666:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83374200/0xe83a7ce271e22563 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc02d expref: -99 pid: 65330 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699669:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486c600/0xe83a7ce271e22524 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbf7e expref: -99 pid: 65324 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699673:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375600/0xe83a7ce271e224de lrc: 1/0,0 mode: --/PR res: [0x240000403:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbeba expref: -99 pid: 65319 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699678:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375c00/0xe83a7ce271e2249f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbe0b expref: -99 pid: 65313 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699682:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83374c00/0xe83a7ce271e22428 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbd47 expref: -99 pid: 65304 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699688:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ff600/0xe83a7ce271e22348 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbc6e expref: -99 pid: 65293 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699692:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375000/0xe83a7ce271e22268 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbb95 expref: -99 pid: 65282 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699696:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ff000/0xe83a7ce271e22150 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbaa7 expref: -99 pid: 65270 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699700:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b72200/0xe83a7ce271e22118 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdba0d expref: -99 pid: 65265 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699704:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b73600/0xe83a7ce271e220e0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdb96c expref: -99 pid: 65260 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699708:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ffe00/0xe83a7ce271e220a8 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdb8d2 expref: -99 pid: 65255 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699712:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b813c8400/0xe83a7ce271e22085 lrc: 1/0,0 mode: --/PR res: [0x240000404:0x6:0x0].0x0 bits 0x10/0x0 rrc: 7 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x557988ac98fdb862 expref: -99 pid: 65254 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1769075478.699715:0:13603:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 36 locks packed 00010000:00010000:1.0:1769075478.699772:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b80030a00/0xe83a7ce271e22eb6 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdd1a6 expref: -99 pid: 65488 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699800:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f79c00/0xe83a7ce271e22e77 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdd0f7 expref: -99 pid: 65483 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699843:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86218c00/0xe83a7ce271e22e38 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdd048 expref: -99 pid: 65478 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699881:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f79a00/0xe83a7ce271e22df2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcf84 expref: -99 pid: 65472 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699889:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486c400/0xe83a7ce271e22db3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdced5 expref: -99 pid: 65467 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699897:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f79800/0xe83a7ce271e22d74 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdce26 expref: -99 pid: 65462 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699903:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229200/0xe83a7ce271e22cfd lrc: 0/0,0 mode: --/PR res: [0x240000403:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcd62 expref: -99 pid: 65453 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699910:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b80030000/0xe83a7ce271e22c1d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcc89 expref: -99 pid: 65443 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699917:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228c00/0xe83a7ce271e22bde lrc: 0/0,0 mode: --/PR res: [0x240000403:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcbda expref: -99 pid: 65438 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699926:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f51800/0xe83a7ce271e22b9f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdcb2b expref: -99 pid: 65433 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699934:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228800/0xe83a7ce271e22b59 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdca67 expref: -99 pid: 65428 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699940:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229a00/0xe83a7ce271e22b1a lrc: 0/0,0 mode: --/PR res: [0x240000403:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc9b8 expref: -99 pid: 65422 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699947:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86218000/0xe83a7ce271e22aa3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc8f4 expref: -99 pid: 65413 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699955:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229800/0xe83a7ce271e229c3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc81b expref: -99 pid: 65402 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699962:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486ce00/0xe83a7ce271e2291b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x54:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc757 expref: -99 pid: 65395 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699970:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229000/0xe83a7ce271e228dc lrc: 0/0,0 mode: --/PR res: [0x240000403:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc6a8 expref: -99 pid: 65389 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699976:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229400/0xe83a7ce271e2289d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc5f9 expref: -99 pid: 65384 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699983:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86219a00/0xe83a7ce271e2285e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc54a expref: -99 pid: 65379 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699989:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228600/0xe83a7ce271e227e7 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc486 expref: -99 pid: 65370 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.699996:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86228e00/0xe83a7ce271e22707 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc3ad expref: -99 pid: 65358 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700002:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486c000/0xe83a7ce271e2265f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc2e9 expref: -99 pid: 65351 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700008:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b86229e00/0xe83a7ce271e22620 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc23a expref: -99 pid: 65346 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700015:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375400/0xe83a7ce271e225e1 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc18b expref: -99 pid: 65341 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700024:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375200/0xe83a7ce271e225a2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc0dc expref: -99 pid: 65335 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700031:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83374200/0xe83a7ce271e22563 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdc02d expref: -99 pid: 65330 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700038:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8486c600/0xe83a7ce271e22524 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbf7e expref: -99 pid: 65324 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700045:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375600/0xe83a7ce271e224de lrc: 0/0,0 mode: --/PR res: [0x240000403:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbeba expref: -99 pid: 65319 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700051:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375c00/0xe83a7ce271e2249f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbe0b expref: -99 pid: 65313 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700059:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83374c00/0xe83a7ce271e22428 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbd47 expref: -99 pid: 65304 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700066:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ff600/0xe83a7ce271e22348 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbc6e expref: -99 pid: 65293 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700074:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83375000/0xe83a7ce271e22268 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbb95 expref: -99 pid: 65282 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700080:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ff000/0xe83a7ce271e22150 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdbaa7 expref: -99 pid: 65270 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700087:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b72200/0xe83a7ce271e22118 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdba0d expref: -99 pid: 65265 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700095:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b73600/0xe83a7ce271e220e0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdb96c expref: -99 pid: 65260 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700103:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ffe00/0xe83a7ce271e220a8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdb8d2 expref: -99 pid: 65255 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075478.700109:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b813c8400/0xe83a7ce271e22085 lrc: 0/0,0 mode: --/PR res: [0x240000404:0x6:0x0].0x0 bits 0x10/0x0 rrc: 7 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x557988ac98fdb862 expref: -99 pid: 65254 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1769075478.700116:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075478.700117:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075478.700119:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075479.016659:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075479.016733:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075479.016739:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075480.040559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075480.040571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075481.064512:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075481.768648:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075482.088519:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075482.088641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075483.112554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075483.112580:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075483.112590:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075483.112595:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075483.112606:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2468 00000100:00080000:1.0:1769075483.112621:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075483.112634:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075483.112686:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616390656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075483.112711:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075483.112714:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075483.112716:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075483.112719:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2468 00000100:00080000:1.0:1769075483.112723:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075483.112727:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075483.112737:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075483.112766:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616390784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075483.112783:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075483.112787:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075483.112788:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075483.112791:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2468 00000100:00080000:1.0:1769075483.112794:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075483.112797:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075483.112801:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075483.112811:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616390912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075483.112818:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075483.112821:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075483.112822:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075483.112826:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2468 00000100:00080000:1.0:1769075483.112828:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075483.112831:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075483.112835:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075483.112847:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616391040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075483.112870:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075483.112874:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075483.112876:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075483.112879:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2468 00000100:00080000:1.0:1769075483.112882:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075483.112885:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075483.112891:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075483.112903:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7b80 x1855007616391168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075483.112942:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075483.112960:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075483.112965:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075483.112984:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2468 00000100:00080000:1.0:1769075483.113002:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075483.113014:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075483.113021:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075483.113037:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6680 x1855007616391296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075483.113064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075483.113067:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075483.113069:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075483.113075:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2468 00000100:00080000:1.0:1769075483.113078:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075483.113082:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075483.113085:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075483.113097:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5180 x1855007616391424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075484.136555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075484.136604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075484.840602:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075485.160503:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075485.160556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075485.160563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075486.184542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075486.184575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075487.208638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075487.912720:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075488.232547:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769075488.232550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075488.232558:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075488.232562:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075488.232571:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2473 00000100:00080000:1.0:1769075488.232581:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075488.232588:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075488.232596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075488.232630:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5180 x1855007616391552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075488.232675:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075488.232679:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075488.232681:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075488.232684:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2473 00000100:00080000:1.0:1769075488.232688:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075488.232692:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075488.232700:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075488.232713:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6680 x1855007616391680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075488.232725:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075488.232727:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075488.232729:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075488.232732:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2473 00000100:00080000:1.0:1769075488.232735:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075488.232738:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075488.232743:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075488.232754:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7b80 x1855007616391808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075488.232767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075488.232770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075488.232772:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075488.232780:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2473 00000100:00080000:1.0:1769075488.232786:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075488.232789:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075488.232793:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075488.232803:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616391936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075488.232810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075488.232813:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075488.232814:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075488.232816:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2473 00000100:00080000:1.0:1769075488.232821:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075488.232827:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075488.232833:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075488.232846:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616392064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075488.232854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075488.232862:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075488.232864:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075488.232872:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2473 00000100:00080000:1.0:1769075488.232875:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075488.232881:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075488.232886:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075488.232897:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616392192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075488.232909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075488.232912:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075488.232913:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075488.232917:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2473 00000100:00080000:1.0:1769075488.232920:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075488.232927:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075488.232930:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075488.232940:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616392320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075488.936626:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 30/30 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1769075488.936715:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 56/56 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1769075488.936754:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 29/85 locks to regular blp list, 2 blwis in pool 00010000:00010000:1.0:1769075488.936793:0:13603:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 29 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1769075488.937059:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075488.937063:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075488.937067:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075488.937087:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d24e00/0xe83a7ce271e247f3 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x48:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdfb44 expref: -99 pid: 65887 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937114:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x99:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937147:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x99:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937162:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f73800/0xe83a7ce271e24705 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x47:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdf9bc expref: -99 pid: 65872 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937181:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x97:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937205:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x97:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937216:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113f800/0xe83a7ce271e245e6 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x46:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdf834 expref: -99 pid: 65856 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937232:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x95:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937251:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x95:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937266:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113e200/0xe83a7ce271e24561 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x45:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdf6c1 expref: -99 pid: 65845 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937284:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x93:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937309:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x93:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937322:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d25200/0xe83a7ce271e244a4 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x44:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdf539 expref: -99 pid: 65830 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937333:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x91:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937354:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x91:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937370:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833afa00/0xe83a7ce271e24315 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdf387 expref: -99 pid: 65809 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937385:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x8f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937414:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x8f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937426:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833ae400/0xe83a7ce271e2422e lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdf214 expref: -99 pid: 65797 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937437:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x8d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937477:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x8d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937486:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833af800/0xe83a7ce271e241b0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x41:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdf0b6 expref: -99 pid: 65786 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937502:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x8b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937525:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x8b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937543:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833af600/0xe83a7ce271e24059 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x40:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdef19 expref: -99 pid: 65768 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937554:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x89:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937580:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x89:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937603:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f06000/0xe83a7ce271e23fdb lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x3f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdedbb expref: -99 pid: 65757 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937617:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x87:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937662:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x87:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937682:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486d400/0xe83a7ce271e23f5d lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x3e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdec5d expref: -99 pid: 65747 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937695:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x85:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937725:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x85:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937745:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8680ec00/0xe83a7ce271e23e3e lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x3d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdead5 expref: -99 pid: 65730 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937797:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x83:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937820:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x83:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937838:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f06200/0xe83a7ce271e23dc0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x3c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fde977 expref: -99 pid: 65720 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937852:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x81:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937877:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x81:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937888:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d24400/0xe83a7ce271e23d0a lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x3b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fde804 expref: -99 pid: 65705 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937908:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x7f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937930:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x7f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937948:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d25c00/0xe83a7ce271e23beb lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x3a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fde67c expref: -99 pid: 65689 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.937960:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x7d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.937978:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x7d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.937991:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8680f600/0xe83a7ce271e23afd lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fde4f4 expref: -99 pid: 65676 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938011:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x7b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938033:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x7b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938053:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8680e000/0xe83a7ce271e239de lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x38:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fde36c expref: -99 pid: 65660 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938075:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x79:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938101:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x79:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938115:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486d200/0xe83a7ce271e238b8 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fde1cf expref: -99 pid: 65643 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938126:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x77:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938149:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x77:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938164:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d25e00/0xe83a7ce271e2382c lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fde047 expref: -99 pid: 65633 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938179:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x75:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938200:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x75:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938213:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486d000/0xe83a7ce271e23776 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdded4 expref: -99 pid: 65618 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938235:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x73:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938266:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x73:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938282:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b86218e00/0xe83a7ce271e235e7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x34:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fddd22 expref: -99 pid: 65598 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938300:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x71:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938327:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x71:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938353:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486de00/0xe83a7ce271e234f9 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x33:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fddb9a expref: -99 pid: 65585 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938384:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x6f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938412:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x6f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938430:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8695a800/0xe83a7ce271e2347b lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x32:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdda3c expref: -99 pid: 65575 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938485:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x6d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938511:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x6d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938530:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8148f000/0xe83a7ce271e23324 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x31:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdd89f expref: -99 pid: 65556 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938558:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x6b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938583:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x6b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938605:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8148f600/0xe83a7ce271e2329f lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x30:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdd72c expref: -99 pid: 65546 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938633:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x69:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938675:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x69:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938695:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b86219000/0xe83a7ce271e2321a lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x2f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdd5b9 expref: -99 pid: 65534 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938718:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x67:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938753:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x67:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938770:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b80030200/0xe83a7ce271e230c3 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x2e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdd41c expref: -99 pid: 65514 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938819:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x65:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938838:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x65:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938852:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f78200/0xe83a7ce271e22f6c lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdd27f expref: -99 pid: 65495 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938874:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x63:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938891:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x63:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000100:00080000:1.0:1769075488.938897:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83504a80 x1855007616392448/t0(0) o103->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 552/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1769075488.938906:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f78a00/0xe83a7ce271e22e85 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fdd10c expref: -99 pid: 65483 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075488.938936:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x61:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075488.938957:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x61:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075488.938985:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d24e00/0xe83a7ce271e247f3 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x48:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdfb44 expref: -99 pid: 65887 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939006:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f73800/0xe83a7ce271e24705 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x47:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf9bc expref: -99 pid: 65872 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939024:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113f800/0xe83a7ce271e245e6 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x46:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf834 expref: -99 pid: 65856 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939037:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113e200/0xe83a7ce271e24561 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x45:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf6c1 expref: -99 pid: 65845 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939051:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d25200/0xe83a7ce271e244a4 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x44:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf539 expref: -99 pid: 65830 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939078:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833afa00/0xe83a7ce271e24315 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf387 expref: -99 pid: 65809 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939099:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833ae400/0xe83a7ce271e2422e lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf214 expref: -99 pid: 65797 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939120:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833af800/0xe83a7ce271e241b0 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x41:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf0b6 expref: -99 pid: 65786 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939129:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833af600/0xe83a7ce271e24059 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x40:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdef19 expref: -99 pid: 65768 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939145:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f06000/0xe83a7ce271e23fdb lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x3f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdedbb expref: -99 pid: 65757 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939165:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486d400/0xe83a7ce271e23f5d lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x3e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdec5d expref: -99 pid: 65747 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939184:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8680ec00/0xe83a7ce271e23e3e lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x3d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdead5 expref: -99 pid: 65730 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939192:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f06200/0xe83a7ce271e23dc0 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x3c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde977 expref: -99 pid: 65720 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939214:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d24400/0xe83a7ce271e23d0a lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x3b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde804 expref: -99 pid: 65705 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939230:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d25c00/0xe83a7ce271e23beb lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x3a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde67c expref: -99 pid: 65689 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939240:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8680f600/0xe83a7ce271e23afd lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde4f4 expref: -99 pid: 65676 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939266:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8680e000/0xe83a7ce271e239de lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x38:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde36c expref: -99 pid: 65660 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939289:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486d200/0xe83a7ce271e238b8 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde1cf expref: -99 pid: 65643 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939311:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d25e00/0xe83a7ce271e2382c lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde047 expref: -99 pid: 65633 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939331:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486d000/0xe83a7ce271e23776 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdded4 expref: -99 pid: 65618 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939342:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b86218e00/0xe83a7ce271e235e7 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x34:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fddd22 expref: -99 pid: 65598 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939360:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486de00/0xe83a7ce271e234f9 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x33:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fddb9a expref: -99 pid: 65585 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939372:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8695a800/0xe83a7ce271e2347b lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x32:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdda3c expref: -99 pid: 65575 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939388:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8148f000/0xe83a7ce271e23324 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x31:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd89f expref: -99 pid: 65556 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939406:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8148f600/0xe83a7ce271e2329f lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x30:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd72c expref: -99 pid: 65546 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939419:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b86219000/0xe83a7ce271e2321a lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x2f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd5b9 expref: -99 pid: 65534 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939431:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b80030200/0xe83a7ce271e230c3 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x2e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd41c expref: -99 pid: 65514 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939451:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f78200/0xe83a7ce271e22f6c lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd27f expref: -99 pid: 65495 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939490:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f78a00/0xe83a7ce271e22e85 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd10c expref: -99 pid: 65483 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939495:0:13603:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 29 locks packed 00010000:00010000:3.0:1769075488.939503:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d24e00/0xe83a7ce271e247f3 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x48:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdfb44 expref: -99 pid: 65887 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939524:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f73800/0xe83a7ce271e24705 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x47:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf9bc expref: -99 pid: 65872 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939544:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113f800/0xe83a7ce271e245e6 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x46:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf834 expref: -99 pid: 65856 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939571:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113e200/0xe83a7ce271e24561 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x45:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf6c1 expref: -99 pid: 65845 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939596:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d25200/0xe83a7ce271e244a4 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x44:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf539 expref: -99 pid: 65830 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939659:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833afa00/0xe83a7ce271e24315 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf387 expref: -99 pid: 65809 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939683:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833ae400/0xe83a7ce271e2422e lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf214 expref: -99 pid: 65797 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939717:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833af800/0xe83a7ce271e241b0 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x41:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdf0b6 expref: -99 pid: 65786 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939754:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b833af600/0xe83a7ce271e24059 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x40:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdef19 expref: -99 pid: 65768 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939788:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f06000/0xe83a7ce271e23fdb lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x3f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdedbb expref: -99 pid: 65757 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939815:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486d400/0xe83a7ce271e23f5d lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x3e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdec5d expref: -99 pid: 65747 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939828:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8680ec00/0xe83a7ce271e23e3e lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x3d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdead5 expref: -99 pid: 65730 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939840:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f06200/0xe83a7ce271e23dc0 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x3c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde977 expref: -99 pid: 65720 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939853:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d24400/0xe83a7ce271e23d0a lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x3b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde804 expref: -99 pid: 65705 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939880:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d25c00/0xe83a7ce271e23beb lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x3a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde67c expref: -99 pid: 65689 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939907:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8680f600/0xe83a7ce271e23afd lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde4f4 expref: -99 pid: 65676 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939939:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8680e000/0xe83a7ce271e239de lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x38:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde36c expref: -99 pid: 65660 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939970:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486d200/0xe83a7ce271e238b8 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde1cf expref: -99 pid: 65643 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.939995:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d25e00/0xe83a7ce271e2382c lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fde047 expref: -99 pid: 65633 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940027:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486d000/0xe83a7ce271e23776 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdded4 expref: -99 pid: 65618 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940059:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b86218e00/0xe83a7ce271e235e7 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x34:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fddd22 expref: -99 pid: 65598 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940090:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8486de00/0xe83a7ce271e234f9 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x33:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fddb9a expref: -99 pid: 65585 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940119:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8695a800/0xe83a7ce271e2347b lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x32:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdda3c expref: -99 pid: 65575 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940143:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8148f000/0xe83a7ce271e23324 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x31:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd89f expref: -99 pid: 65556 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940175:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8148f600/0xe83a7ce271e2329f lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x30:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd72c expref: -99 pid: 65546 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940198:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b86219000/0xe83a7ce271e2321a lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x2f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd5b9 expref: -99 pid: 65534 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940222:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b80030200/0xe83a7ce271e230c3 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x2e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd41c expref: -99 pid: 65514 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940257:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f78200/0xe83a7ce271e22f6c lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd27f expref: -99 pid: 65495 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940282:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f78a00/0xe83a7ce271e22e85 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fdd10c expref: -99 pid: 65483 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075488.940310:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075488.940313:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075488.940316:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075488.940330:0:2420:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b84558380 x1855007616392704/t0(0) o103->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 544/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075488.940357:0:2420:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1769075488.940364:0:2420:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b85ac9f80 x1855007616392576/t0(0) o103->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 760/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075489.256551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075489.256558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075489.256596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075489.960625:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075490.280574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075490.280575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075491.304540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075491.304554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:3.0:1769075492.008554:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075492.328541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075492.328543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075493.352561:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075493.352568:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075493.352590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075493.352594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075493.352602:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2479 00000100:00080000:1.0:1769075493.352613:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075493.352621:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075493.352657:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616392832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075493.352679:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075493.352682:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075493.352683:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075493.352686:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2479 00000100:00080000:1.0:1769075493.352689:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075493.352692:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075493.352702:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075493.352713:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616392960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075493.352723:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075493.352725:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075493.352727:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075493.352730:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2479 00000100:00080000:1.0:1769075493.352733:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075493.352735:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075493.352738:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075493.352748:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616393088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075493.352754:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075493.352757:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075493.352758:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075493.352761:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2479 00000100:00080000:1.0:1769075493.352763:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075493.352765:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075493.352769:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075493.352778:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616393216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075493.352786:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075493.352788:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075493.352790:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075493.352793:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2479 00000100:00080000:1.0:1769075493.352795:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075493.352799:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075493.352802:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075493.352811:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616393344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075493.352837:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075493.352839:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075493.352841:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075493.352846:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2479 00000100:00080000:1.0:1769075493.352851:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075493.352857:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075493.352863:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075493.352873:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616393472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075493.352886:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075493.352889:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075493.352890:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075493.352894:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2479 00000100:00080000:1.0:1769075493.352897:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075493.352899:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075493.352901:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075493.352911:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616393600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075494.056627:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075494.376553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075494.376601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075495.400534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:3.0:1769075496.104594:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075496.424687:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075496.424721:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075497.448565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075497.448567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075497.448628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075498.152581:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075498.472574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075498.472574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075499.176599:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 16/16 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1769075499.176690:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 56/56 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1769075499.176735:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 16/72 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1769075499.176758:0:13603:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 56 locks of 16 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1769075499.176768:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621be00/0xe83a7ce271e26130 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2616 expref: -99 pid: 66291 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075499.176789:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1769075499.176810:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd2:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.176850:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a600/0xe83a7ce271e260b9 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2552 expref: -99 pid: 66284 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.176872:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd1:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.176899:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476e00/0xe83a7ce271e26042 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe248e expref: -99 pid: 66275 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.176917:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd0:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.176935:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85202c00/0xe83a7ce271e25f9a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xcf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe23ca expref: -99 pid: 66267 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.176963:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xcf:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.176984:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621ba00/0xe83a7ce271e25f5b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe231b expref: -99 pid: 66262 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177016:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xce:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177043:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f06400/0xe83a7ce271e25ee4 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2257 expref: -99 pid: 66255 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177065:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xcd:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177088:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a400/0xe83a7ce271e25ea5 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe21a8 expref: -99 pid: 66250 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177110:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xcc:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177130:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a000/0xe83a7ce271e25e66 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe20f9 expref: -99 pid: 66244 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177163:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xcb:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177185:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a200/0xe83a7ce271e25def lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2035 expref: -99 pid: 66235 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177212:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xca:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177237:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203000/0xe83a7ce271e25d40 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1f5c expref: -99 pid: 66228 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177251:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc9:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177275:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9230a000/0xe83a7ce271e25cc9 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1e98 expref: -99 pid: 66221 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177303:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc8:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177327:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206b800/0xe83a7ce271e25c8a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1de9 expref: -99 pid: 66215 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177357:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc7:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177376:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9230ac00/0xe83a7ce271e25c4b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1d3a expref: -99 pid: 66210 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177394:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc6:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177412:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206b600/0xe83a7ce271e25bd4 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1c76 expref: -99 pid: 66201 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177427:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc5:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177446:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206ac00/0xe83a7ce271e25b2c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1bb2 expref: -99 pid: 66194 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177511:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc4:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177538:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76000/0xe83a7ce271e25ab5 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1aee expref: -99 pid: 66184 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177565:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc3:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177590:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546600/0xe83a7ce271e25a06 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1a15 expref: -99 pid: 66177 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177605:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc2:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177630:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83547000/0xe83a7ce271e259c0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1951 expref: -99 pid: 66172 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177656:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc1:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177674:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07000/0xe83a7ce271e25981 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe18a2 expref: -99 pid: 66167 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177703:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xc0:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177729:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07200/0xe83a7ce271e25942 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe17f3 expref: -99 pid: 66161 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177742:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xbf:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177758:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83547c00/0xe83a7ce271e258cb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe172f expref: -99 pid: 66152 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177773:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xbe:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177789:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546200/0xe83a7ce271e25823 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe166b expref: -99 pid: 66145 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177802:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xbd:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177823:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07400/0xe83a7ce271e257ac lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe15a7 expref: -99 pid: 66136 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177849:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xbc:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177869:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92068600/0xe83a7ce271e256fd lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe14ce expref: -99 pid: 66128 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177897:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xbb:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177915:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069000/0xe83a7ce271e256be lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe141f expref: -99 pid: 66123 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177946:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xba:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.177964:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8695ae00/0xe83a7ce271e25647 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe135b expref: -99 pid: 66114 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.177998:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb9:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178015:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b73800/0xe83a7ce271e25598 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1282 expref: -99 pid: 66107 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178046:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb8:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178064:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069e00/0xe83a7ce271e25521 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe11be expref: -99 pid: 66097 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178091:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb7:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178119:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069400/0xe83a7ce271e25479 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe10fa expref: -99 pid: 66089 timeout: 0 lvb_type: 3 00000100:00080000:1.0:1769075499.178120:0:2418:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83507b80 x1855007616394624/t0(0) o103->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 440/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000080:00200000:3.0:1769075499.178152:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb6:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178189:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92068200/0xe83a7ce271e25433 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe1036 expref: -99 pid: 66084 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178217:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb5:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178240:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92068800/0xe83a7ce271e253f4 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0f87 expref: -99 pid: 66079 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178253:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb4:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178271:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8695b800/0xe83a7ce271e253b5 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0ed8 expref: -99 pid: 66073 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178296:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb3:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178313:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069200/0xe83a7ce271e25376 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0e29 expref: -99 pid: 66068 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178329:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb2:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178353:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85202600/0xe83a7ce271e25337 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0d7a expref: -99 pid: 66063 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178377:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb1:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178408:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069c00/0xe83a7ce271e252c0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0cb6 expref: -99 pid: 66054 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178439:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xb0:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178500:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b72e00/0xe83a7ce271e251e0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xaf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0bdd expref: -99 pid: 66044 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178531:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xaf:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178551:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551a200/0xe83a7ce271e25169 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0b19 expref: -99 pid: 66035 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178578:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xae:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178603:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07c00/0xe83a7ce271e25089 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xad:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0a40 expref: -99 pid: 66024 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178620:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xad:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178636:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76400/0xe83a7ce271e24fe1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe097c expref: -99 pid: 66017 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178664:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xac:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178683:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76c00/0xe83a7ce271e24f6a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xab:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe08b8 expref: -99 pid: 66009 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178694:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xab:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178705:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85202400/0xe83a7ce271e24f2b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0809 expref: -99 pid: 66004 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178714:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xaa:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178724:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203800/0xe83a7ce271e24eec lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe075a expref: -99 pid: 65999 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178733:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa9:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178743:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76800/0xe83a7ce271e24e75 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0696 expref: -99 pid: 65992 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178753:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa8:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178803:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76200/0xe83a7ce271e24dfe lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe05d2 expref: -99 pid: 65984 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178814:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa7:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178826:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f77600/0xe83a7ce271e24db8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe050e expref: -99 pid: 65979 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178835:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa6:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178846:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d25800/0xe83a7ce271e24d79 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe045f expref: -99 pid: 65974 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178857:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa5:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178868:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f77200/0xe83a7ce271e24d02 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe039b expref: -99 pid: 65967 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178878:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa4:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178888:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113e800/0xe83a7ce271e24c8b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe02d7 expref: -99 pid: 65959 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178897:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa3:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178907:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8695ba00/0xe83a7ce271e24c14 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe0213 expref: -99 pid: 65950 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178916:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa2:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178926:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113e000/0xe83a7ce271e24b6c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe014f expref: -99 pid: 65943 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178935:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa1:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178945:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113fe00/0xe83a7ce271e24af5 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe008b expref: -99 pid: 65936 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178954:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xa0:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178964:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551b200/0xe83a7ce271e24a7e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdffc7 expref: -99 pid: 65928 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178973:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x9f:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.178982:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f77e00/0xe83a7ce271e24a38 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdff03 expref: -99 pid: 65923 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.178992:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x9e:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.179002:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551b600/0xe83a7ce271e249c1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x9d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdfe3f expref: -99 pid: 65914 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.179011:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x9d:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.179021:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f72400/0xe83a7ce271e248e1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdfd66 expref: -99 pid: 65905 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.179031:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x9c:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.179041:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113ec00/0xe83a7ce271e2486a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fdfca2 expref: -99 pid: 65897 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075499.179050:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x9b:0x0]: losing layout lock 00010000:00010000:3.0:1769075499.179071:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621be00/0xe83a7ce271e26130 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2616 expref: -99 pid: 66291 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179079:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a600/0xe83a7ce271e260b9 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2552 expref: -99 pid: 66284 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179086:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476e00/0xe83a7ce271e26042 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe248e expref: -99 pid: 66275 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179094:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85202c00/0xe83a7ce271e25f9a lrc: 1/0,0 mode: --/PR res: [0x240000403:0xcf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe23ca expref: -99 pid: 66267 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179107:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621ba00/0xe83a7ce271e25f5b lrc: 1/0,0 mode: --/PR res: [0x240000403:0xce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe231b expref: -99 pid: 66262 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179115:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f06400/0xe83a7ce271e25ee4 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2257 expref: -99 pid: 66255 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179122:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a400/0xe83a7ce271e25ea5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe21a8 expref: -99 pid: 66250 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179130:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a000/0xe83a7ce271e25e66 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe20f9 expref: -99 pid: 66244 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179138:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a200/0xe83a7ce271e25def lrc: 1/0,0 mode: --/PR res: [0x240000403:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2035 expref: -99 pid: 66235 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179145:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203000/0xe83a7ce271e25d40 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1f5c expref: -99 pid: 66228 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179151:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9230a000/0xe83a7ce271e25cc9 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1e98 expref: -99 pid: 66221 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179159:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206b800/0xe83a7ce271e25c8a lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1de9 expref: -99 pid: 66215 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179167:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9230ac00/0xe83a7ce271e25c4b lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1d3a expref: -99 pid: 66210 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179174:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206b600/0xe83a7ce271e25bd4 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1c76 expref: -99 pid: 66201 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179187:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206ac00/0xe83a7ce271e25b2c lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1bb2 expref: -99 pid: 66194 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179195:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76000/0xe83a7ce271e25ab5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1aee expref: -99 pid: 66184 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179202:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546600/0xe83a7ce271e25a06 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1a15 expref: -99 pid: 66177 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179210:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83547000/0xe83a7ce271e259c0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1951 expref: -99 pid: 66172 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179217:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07000/0xe83a7ce271e25981 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe18a2 expref: -99 pid: 66167 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179225:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07200/0xe83a7ce271e25942 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe17f3 expref: -99 pid: 66161 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179232:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83547c00/0xe83a7ce271e258cb lrc: 1/0,0 mode: --/PR res: [0x240000403:0xbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe172f expref: -99 pid: 66152 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179239:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546200/0xe83a7ce271e25823 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe166b expref: -99 pid: 66145 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179247:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07400/0xe83a7ce271e257ac lrc: 1/0,0 mode: --/PR res: [0x240000403:0xbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe15a7 expref: -99 pid: 66136 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179254:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92068600/0xe83a7ce271e256fd lrc: 1/0,0 mode: --/PR res: [0x240000403:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe14ce expref: -99 pid: 66128 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179266:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069000/0xe83a7ce271e256be lrc: 1/0,0 mode: --/PR res: [0x240000403:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe141f expref: -99 pid: 66123 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179275:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8695ae00/0xe83a7ce271e25647 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe135b expref: -99 pid: 66114 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179282:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b73800/0xe83a7ce271e25598 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1282 expref: -99 pid: 66107 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179289:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069e00/0xe83a7ce271e25521 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe11be expref: -99 pid: 66097 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179296:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069400/0xe83a7ce271e25479 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe10fa expref: -99 pid: 66089 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179303:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92068200/0xe83a7ce271e25433 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1036 expref: -99 pid: 66084 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179310:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92068800/0xe83a7ce271e253f4 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0f87 expref: -99 pid: 66079 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179317:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8695b800/0xe83a7ce271e253b5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0ed8 expref: -99 pid: 66073 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179324:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069200/0xe83a7ce271e25376 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0e29 expref: -99 pid: 66068 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179332:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85202600/0xe83a7ce271e25337 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0d7a expref: -99 pid: 66063 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179344:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069c00/0xe83a7ce271e252c0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0cb6 expref: -99 pid: 66054 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179352:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b72e00/0xe83a7ce271e251e0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xaf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0bdd expref: -99 pid: 66044 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179360:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551a200/0xe83a7ce271e25169 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0b19 expref: -99 pid: 66035 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179366:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07c00/0xe83a7ce271e25089 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xad:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0a40 expref: -99 pid: 66024 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179373:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76400/0xe83a7ce271e24fe1 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe097c expref: -99 pid: 66017 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179380:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76c00/0xe83a7ce271e24f6a lrc: 1/0,0 mode: --/PR res: [0x240000403:0xab:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe08b8 expref: -99 pid: 66009 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179387:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85202400/0xe83a7ce271e24f2b lrc: 1/0,0 mode: --/PR res: [0x240000403:0xaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0809 expref: -99 pid: 66004 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179395:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203800/0xe83a7ce271e24eec lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe075a expref: -99 pid: 65999 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179402:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76800/0xe83a7ce271e24e75 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0696 expref: -99 pid: 65992 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179410:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76200/0xe83a7ce271e24dfe lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe05d2 expref: -99 pid: 65984 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179423:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f77600/0xe83a7ce271e24db8 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe050e expref: -99 pid: 65979 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179430:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d25800/0xe83a7ce271e24d79 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe045f expref: -99 pid: 65974 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179437:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f77200/0xe83a7ce271e24d02 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe039b expref: -99 pid: 65967 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179445:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113e800/0xe83a7ce271e24c8b lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe02d7 expref: -99 pid: 65959 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179490:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8695ba00/0xe83a7ce271e24c14 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0213 expref: -99 pid: 65950 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179500:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113e000/0xe83a7ce271e24b6c lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe014f expref: -99 pid: 65943 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179507:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113fe00/0xe83a7ce271e24af5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe008b expref: -99 pid: 65936 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179515:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551b200/0xe83a7ce271e24a7e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdffc7 expref: -99 pid: 65928 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179522:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f77e00/0xe83a7ce271e24a38 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdff03 expref: -99 pid: 65923 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179529:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551b600/0xe83a7ce271e249c1 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x9d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdfe3f expref: -99 pid: 65914 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179541:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f72400/0xe83a7ce271e248e1 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdfd66 expref: -99 pid: 65905 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179548:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113ec00/0xe83a7ce271e2486a lrc: 1/0,0 mode: --/PR res: [0x240000403:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdfca2 expref: -99 pid: 65897 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179555:0:13603:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 56 locks packed 00010000:00010000:3.0:1769075499.179571:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621be00/0xe83a7ce271e26130 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2616 expref: -99 pid: 66291 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179591:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a600/0xe83a7ce271e260b9 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2552 expref: -99 pid: 66284 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179610:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476e00/0xe83a7ce271e26042 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe248e expref: -99 pid: 66275 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179625:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85202c00/0xe83a7ce271e25f9a lrc: 0/0,0 mode: --/PR res: [0x240000403:0xcf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe23ca expref: -99 pid: 66267 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179636:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621ba00/0xe83a7ce271e25f5b lrc: 0/0,0 mode: --/PR res: [0x240000403:0xce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe231b expref: -99 pid: 66262 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179647:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f06400/0xe83a7ce271e25ee4 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2257 expref: -99 pid: 66255 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179658:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a400/0xe83a7ce271e25ea5 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe21a8 expref: -99 pid: 66250 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179668:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a000/0xe83a7ce271e25e66 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe20f9 expref: -99 pid: 66244 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179680:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621a200/0xe83a7ce271e25def lrc: 0/0,0 mode: --/PR res: [0x240000403:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2035 expref: -99 pid: 66235 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179692:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203000/0xe83a7ce271e25d40 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1f5c expref: -99 pid: 66228 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179703:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9230a000/0xe83a7ce271e25cc9 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1e98 expref: -99 pid: 66221 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179714:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206b800/0xe83a7ce271e25c8a lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1de9 expref: -99 pid: 66215 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179726:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9230ac00/0xe83a7ce271e25c4b lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1d3a expref: -99 pid: 66210 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179736:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206b600/0xe83a7ce271e25bd4 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1c76 expref: -99 pid: 66201 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179746:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206ac00/0xe83a7ce271e25b2c lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1bb2 expref: -99 pid: 66194 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179757:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76000/0xe83a7ce271e25ab5 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1aee expref: -99 pid: 66184 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179768:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546600/0xe83a7ce271e25a06 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1a15 expref: -99 pid: 66177 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179778:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83547000/0xe83a7ce271e259c0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1951 expref: -99 pid: 66172 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179789:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07000/0xe83a7ce271e25981 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe18a2 expref: -99 pid: 66167 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179800:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07200/0xe83a7ce271e25942 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe17f3 expref: -99 pid: 66161 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179812:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83547c00/0xe83a7ce271e258cb lrc: 0/0,0 mode: --/PR res: [0x240000403:0xbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe172f expref: -99 pid: 66152 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179822:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546200/0xe83a7ce271e25823 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe166b expref: -99 pid: 66145 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179833:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07400/0xe83a7ce271e257ac lrc: 0/0,0 mode: --/PR res: [0x240000403:0xbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe15a7 expref: -99 pid: 66136 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179843:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92068600/0xe83a7ce271e256fd lrc: 0/0,0 mode: --/PR res: [0x240000403:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe14ce expref: -99 pid: 66128 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179855:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069000/0xe83a7ce271e256be lrc: 0/0,0 mode: --/PR res: [0x240000403:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe141f expref: -99 pid: 66123 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179867:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8695ae00/0xe83a7ce271e25647 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe135b expref: -99 pid: 66114 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179878:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b73800/0xe83a7ce271e25598 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1282 expref: -99 pid: 66107 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179890:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069e00/0xe83a7ce271e25521 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe11be expref: -99 pid: 66097 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179901:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069400/0xe83a7ce271e25479 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe10fa expref: -99 pid: 66089 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179912:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92068200/0xe83a7ce271e25433 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe1036 expref: -99 pid: 66084 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179922:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92068800/0xe83a7ce271e253f4 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0f87 expref: -99 pid: 66079 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179932:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8695b800/0xe83a7ce271e253b5 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0ed8 expref: -99 pid: 66073 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179942:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069200/0xe83a7ce271e25376 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0e29 expref: -99 pid: 66068 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179953:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85202600/0xe83a7ce271e25337 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0d7a expref: -99 pid: 66063 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179963:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92069c00/0xe83a7ce271e252c0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0cb6 expref: -99 pid: 66054 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179974:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85b72e00/0xe83a7ce271e251e0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xaf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0bdd expref: -99 pid: 66044 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179985:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551a200/0xe83a7ce271e25169 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0b19 expref: -99 pid: 66035 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.179996:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f07c00/0xe83a7ce271e25089 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xad:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0a40 expref: -99 pid: 66024 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180007:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76400/0xe83a7ce271e24fe1 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe097c expref: -99 pid: 66017 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180018:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76c00/0xe83a7ce271e24f6a lrc: 0/0,0 mode: --/PR res: [0x240000403:0xab:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe08b8 expref: -99 pid: 66009 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180029:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85202400/0xe83a7ce271e24f2b lrc: 0/0,0 mode: --/PR res: [0x240000403:0xaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0809 expref: -99 pid: 66004 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180041:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203800/0xe83a7ce271e24eec lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe075a expref: -99 pid: 65999 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180051:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76800/0xe83a7ce271e24e75 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0696 expref: -99 pid: 65992 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180062:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f76200/0xe83a7ce271e24dfe lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe05d2 expref: -99 pid: 65984 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180073:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f77600/0xe83a7ce271e24db8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe050e expref: -99 pid: 65979 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180083:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d25800/0xe83a7ce271e24d79 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe045f expref: -99 pid: 65974 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180094:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f77200/0xe83a7ce271e24d02 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe039b expref: -99 pid: 65967 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180105:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113e800/0xe83a7ce271e24c8b lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe02d7 expref: -99 pid: 65959 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180116:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8695ba00/0xe83a7ce271e24c14 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe0213 expref: -99 pid: 65950 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180127:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113e000/0xe83a7ce271e24b6c lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe014f expref: -99 pid: 65943 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180139:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113fe00/0xe83a7ce271e24af5 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe008b expref: -99 pid: 65936 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180151:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551b200/0xe83a7ce271e24a7e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdffc7 expref: -99 pid: 65928 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180161:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f77e00/0xe83a7ce271e24a38 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdff03 expref: -99 pid: 65923 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180172:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551b600/0xe83a7ce271e249c1 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x9d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdfe3f expref: -99 pid: 65914 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180183:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f72400/0xe83a7ce271e248e1 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdfd66 expref: -99 pid: 65905 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180194:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113ec00/0xe83a7ce271e2486a lrc: 0/0,0 mode: --/PR res: [0x240000403:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fdfca2 expref: -99 pid: 65897 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075499.180206:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075499.180208:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075499.180210:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075499.496552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075499.496587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075499.496600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075500.200578:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075500.520562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075501.544538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075501.544547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075502.568546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075502.568548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075502.568606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075503.272600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075503.592594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075503.592653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075503.592673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075503.592713:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075504.296753:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075504.616608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075504.616609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075505.640587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075505.640587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075505.640670:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075506.664597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075506.664610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075506.664650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075507.368674:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075507.688663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075507.688682:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075508.392568:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075508.712543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075508.712544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075509.416637:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 63/63 locks to regular blp list, 1 blwis in pool 00000080:00200000:0.0:1769075509.417759:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xe8:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417767:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d22200/0xe83a7ce271e26af3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe362a expref: -99 pid: 66444 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417773:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xe7:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417781:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476800/0xe83a7ce271e26a7c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe3566 expref: -99 pid: 66437 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417787:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xe6:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417793:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476600/0xe83a7ce271e26a3d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe34b7 expref: -99 pid: 66432 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417799:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xe5:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417805:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476c00/0xe83a7ce271e269f7 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe33f3 expref: -99 pid: 66427 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417811:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xe4:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417818:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d22e00/0xe83a7ce271e269b8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe3344 expref: -99 pid: 66421 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417823:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xe3:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417829:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23e00/0xe83a7ce271e26941 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe3280 expref: -99 pid: 66414 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417834:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xe2:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417840:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23800/0xe83a7ce271e268ca lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe31bc expref: -99 pid: 66405 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417846:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xe1:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417863:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206a000/0xe83a7ce271e267ea lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe30e3 expref: -99 pid: 66393 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417869:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xe0:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417893:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23200/0xe83a7ce271e2670a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe300a expref: -99 pid: 66381 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417904:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xdf:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417915:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206b200/0xe83a7ce271e26662 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2f46 expref: -99 pid: 66374 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417924:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xde:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417934:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23000/0xe83a7ce271e2661c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2e82 expref: -99 pid: 66369 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417959:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xdd:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.417972:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9230b400/0xe83a7ce271e265dd lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2dd3 expref: -99 pid: 66364 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.417993:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xdc:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418004:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2000/0xe83a7ce271e2659e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2d24 expref: -99 pid: 66358 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.418014:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xdb:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418025:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8625ce00/0xe83a7ce271e2655f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2c75 expref: -99 pid: 66353 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.418035:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xda:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418046:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b855c3200/0xe83a7ce271e264e8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2bb1 expref: -99 pid: 66344 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.418056:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd9:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418067:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8625de00/0xe83a7ce271e26439 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2ad8 expref: -99 pid: 66337 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.418076:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd8:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418088:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2600/0xe83a7ce271e263f3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2a14 expref: -99 pid: 66331 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.418098:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd7:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418118:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81477200/0xe83a7ce271e2637c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2950 expref: -99 pid: 66322 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.418144:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd6:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418155:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee3000/0xe83a7ce271e2629c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe2877 expref: -99 pid: 66313 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.418165:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd5:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418175:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2400/0xe83a7ce271e26256 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe27b3 expref: -99 pid: 66308 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.418185:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd4:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418226:0:6177:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee3400/0xe83a7ce271e261df lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe26ef expref: -99 pid: 66298 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075509.418236:0:6177:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0xd3:0x0]: losing layout lock 00010000:00010000:0.0:1769075509.418284:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113a000/0xe83a7ce271e27c50 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x111:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5502 expref: -99 pid: 66730 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418294:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113b000/0xe83a7ce271e27ba1 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x110:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5429 expref: -99 pid: 66723 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418302:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206e200/0xe83a7ce271e27b62 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x10f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe537a expref: -99 pid: 66717 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418341:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113be00/0xe83a7ce271e27aeb lrc: 1/0,0 mode: --/PR res: [0x240000403:0x10e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe52b6 expref: -99 pid: 66710 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418385:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0d400/0xe83a7ce271e27aac lrc: 1/0,0 mode: --/PR res: [0x240000403:0x10d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5207 expref: -99 pid: 66704 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418395:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560e00/0xe83a7ce271e27a6d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x10c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5158 expref: -99 pid: 66699 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418405:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8625d000/0xe83a7ce271e279f6 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x10b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5094 expref: -99 pid: 66691 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418417:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84747000/0xe83a7ce271e2797f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x10a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4fd0 expref: -99 pid: 66684 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418426:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322c00/0xe83a7ce271e27908 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4f0c expref: -99 pid: 66677 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418433:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561800/0xe83a7ce271e27891 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4e48 expref: -99 pid: 66668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418439:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322800/0xe83a7ce271e277e9 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4d84 expref: -99 pid: 66660 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418445:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81323c00/0xe83a7ce271e277aa lrc: 1/0,0 mode: --/PR res: [0x240000403:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4cd5 expref: -99 pid: 66655 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418450:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81323200/0xe83a7ce271e2776b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4c26 expref: -99 pid: 66650 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418456:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560a00/0xe83a7ce271e2772c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4b77 expref: -99 pid: 66645 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418526:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560000/0xe83a7ce271e276ed lrc: 1/0,0 mode: --/PR res: [0x240000403:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4ac8 expref: -99 pid: 66639 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418535:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560400/0xe83a7ce271e276ae lrc: 1/0,0 mode: --/PR res: [0x240000403:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4a19 expref: -99 pid: 66634 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418543:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322000/0xe83a7ce271e2766f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe496a expref: -99 pid: 66629 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418568:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81323400/0xe83a7ce271e27629 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe48a6 expref: -99 pid: 66624 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418576:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322e00/0xe83a7ce271e275b2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe47e2 expref: -99 pid: 66614 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418584:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203a00/0xe83a7ce271e274d2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4709 expref: -99 pid: 66603 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418591:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2200/0xe83a7ce271e273f2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4630 expref: -99 pid: 66592 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418599:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746200/0xe83a7ce271e2730b lrc: 1/0,0 mode: --/PR res: [0x240000403:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4542 expref: -99 pid: 66581 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418607:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621ac00/0xe83a7ce271e27263 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe447e expref: -99 pid: 66573 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418631:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206f600/0xe83a7ce271e27224 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe43cf expref: -99 pid: 66568 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418642:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206f800/0xe83a7ce271e271e5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4320 expref: -99 pid: 66563 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418648:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0c800/0xe83a7ce271e271a6 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4271 expref: -99 pid: 66558 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418654:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746a00/0xe83a7ce271e27167 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe41c2 expref: -99 pid: 66552 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418660:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84747400/0xe83a7ce271e27128 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4113 expref: -99 pid: 66547 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418666:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83547800/0xe83a7ce271e270e9 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4064 expref: -99 pid: 66542 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418672:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746c00/0xe83a7ce271e270aa lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3fb5 expref: -99 pid: 66537 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418694:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546000/0xe83a7ce271e2706b lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3f06 expref: -99 pid: 66531 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418698:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84747c00/0xe83a7ce271e2702c lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3e57 expref: -99 pid: 66526 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418702:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546a00/0xe83a7ce271e26fb5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3d93 expref: -99 pid: 66517 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418706:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206e800/0xe83a7ce271e26f0d lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3ccf expref: -99 pid: 66510 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418713:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746400/0xe83a7ce271e26e96 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3c0b expref: -99 pid: 66500 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418717:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206fa00/0xe83a7ce271e26db6 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3b32 expref: -99 pid: 66489 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418721:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d22c00/0xe83a7ce271e26d0e lrc: 1/0,0 mode: --/PR res: [0x240000403:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3a6e expref: -99 pid: 66482 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418725:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23c00/0xe83a7ce271e26c97 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe39aa expref: -99 pid: 66473 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418729:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23a00/0xe83a7ce271e26bef lrc: 1/0,0 mode: --/PR res: [0x240000403:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe38e6 expref: -99 pid: 66465 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418733:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f72e00/0xe83a7ce271e26bb0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3837 expref: -99 pid: 66460 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418737:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f73600/0xe83a7ce271e26b71 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3788 expref: -99 pid: 66455 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418741:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f73a00/0xe83a7ce271e26b32 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe36d9 expref: -99 pid: 66450 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418745:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d22200/0xe83a7ce271e26af3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe362a expref: -99 pid: 66444 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418749:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476800/0xe83a7ce271e26a7c lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3566 expref: -99 pid: 66437 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418756:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476600/0xe83a7ce271e26a3d lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe34b7 expref: -99 pid: 66432 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418760:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476c00/0xe83a7ce271e269f7 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe33f3 expref: -99 pid: 66427 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418764:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d22e00/0xe83a7ce271e269b8 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3344 expref: -99 pid: 66421 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418768:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23e00/0xe83a7ce271e26941 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3280 expref: -99 pid: 66414 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418772:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23800/0xe83a7ce271e268ca lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe31bc expref: -99 pid: 66405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418776:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206a000/0xe83a7ce271e267ea lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe30e3 expref: -99 pid: 66393 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418780:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23200/0xe83a7ce271e2670a lrc: 1/0,0 mode: --/PR res: [0x240000403:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe300a expref: -99 pid: 66381 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418784:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206b200/0xe83a7ce271e26662 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2f46 expref: -99 pid: 66374 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418788:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23000/0xe83a7ce271e2661c lrc: 1/0,0 mode: --/PR res: [0x240000403:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2e82 expref: -99 pid: 66369 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418792:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9230b400/0xe83a7ce271e265dd lrc: 1/0,0 mode: --/PR res: [0x240000403:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2dd3 expref: -99 pid: 66364 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418799:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2000/0xe83a7ce271e2659e lrc: 1/0,0 mode: --/PR res: [0x240000403:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2d24 expref: -99 pid: 66358 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418803:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8625ce00/0xe83a7ce271e2655f lrc: 1/0,0 mode: --/PR res: [0x240000403:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2c75 expref: -99 pid: 66353 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418807:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b855c3200/0xe83a7ce271e264e8 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2bb1 expref: -99 pid: 66344 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418811:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8625de00/0xe83a7ce271e26439 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2ad8 expref: -99 pid: 66337 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418815:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2600/0xe83a7ce271e263f3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2a14 expref: -99 pid: 66331 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418819:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81477200/0xe83a7ce271e2637c lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2950 expref: -99 pid: 66322 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418823:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee3000/0xe83a7ce271e2629c lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2877 expref: -99 pid: 66313 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418826:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2400/0xe83a7ce271e26256 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe27b3 expref: -99 pid: 66308 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418830:0:6177:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee3400/0xe83a7ce271e261df lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe26ef expref: -99 pid: 66298 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418834:0:6177:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 63 locks packed 00010000:00010000:0.0:1769075509.418854:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113a000/0xe83a7ce271e27c50 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x111:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5502 expref: -99 pid: 66730 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418887:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113b000/0xe83a7ce271e27ba1 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x110:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5429 expref: -99 pid: 66723 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418897:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206e200/0xe83a7ce271e27b62 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x10f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe537a expref: -99 pid: 66717 timeout: 0 lvb_type: 3 00000100:00080000:1.0:1769075509.418905:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83505c00 x1855007616396800/t0(0) o103->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 816/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1769075509.418908:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113be00/0xe83a7ce271e27aeb lrc: 0/0,0 mode: --/PR res: [0x240000403:0x10e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe52b6 expref: -99 pid: 66710 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418964:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0d400/0xe83a7ce271e27aac lrc: 0/0,0 mode: --/PR res: [0x240000403:0x10d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5207 expref: -99 pid: 66704 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.418995:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560e00/0xe83a7ce271e27a6d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x10c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5158 expref: -99 pid: 66699 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419009:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8625d000/0xe83a7ce271e279f6 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x10b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5094 expref: -99 pid: 66691 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419025:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84747000/0xe83a7ce271e2797f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x10a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4fd0 expref: -99 pid: 66684 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419040:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322c00/0xe83a7ce271e27908 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4f0c expref: -99 pid: 66677 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419052:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561800/0xe83a7ce271e27891 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4e48 expref: -99 pid: 66668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419076:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322800/0xe83a7ce271e277e9 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4d84 expref: -99 pid: 66660 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419089:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81323c00/0xe83a7ce271e277aa lrc: 0/0,0 mode: --/PR res: [0x240000403:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4cd5 expref: -99 pid: 66655 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419101:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81323200/0xe83a7ce271e2776b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4c26 expref: -99 pid: 66650 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419111:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560a00/0xe83a7ce271e2772c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4b77 expref: -99 pid: 66645 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419122:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560000/0xe83a7ce271e276ed lrc: 0/0,0 mode: --/PR res: [0x240000403:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4ac8 expref: -99 pid: 66639 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419142:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560400/0xe83a7ce271e276ae lrc: 0/0,0 mode: --/PR res: [0x240000403:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4a19 expref: -99 pid: 66634 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419153:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322000/0xe83a7ce271e2766f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe496a expref: -99 pid: 66629 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419163:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81323400/0xe83a7ce271e27629 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe48a6 expref: -99 pid: 66624 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419174:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322e00/0xe83a7ce271e275b2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe47e2 expref: -99 pid: 66614 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419186:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203a00/0xe83a7ce271e274d2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4709 expref: -99 pid: 66603 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419198:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2200/0xe83a7ce271e273f2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4630 expref: -99 pid: 66592 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419223:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746200/0xe83a7ce271e2730b lrc: 0/0,0 mode: --/PR res: [0x240000403:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4542 expref: -99 pid: 66581 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419235:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621ac00/0xe83a7ce271e27263 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe447e expref: -99 pid: 66573 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419247:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206f600/0xe83a7ce271e27224 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe43cf expref: -99 pid: 66568 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419259:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206f800/0xe83a7ce271e271e5 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4320 expref: -99 pid: 66563 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419282:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0c800/0xe83a7ce271e271a6 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4271 expref: -99 pid: 66558 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419293:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746a00/0xe83a7ce271e27167 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe41c2 expref: -99 pid: 66552 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419332:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84747400/0xe83a7ce271e27128 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4113 expref: -99 pid: 66547 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419377:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83547800/0xe83a7ce271e270e9 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe4064 expref: -99 pid: 66542 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419388:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746c00/0xe83a7ce271e270aa lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3fb5 expref: -99 pid: 66537 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419397:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546000/0xe83a7ce271e2706b lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3f06 expref: -99 pid: 66531 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419405:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84747c00/0xe83a7ce271e2702c lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3e57 expref: -99 pid: 66526 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419411:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83546a00/0xe83a7ce271e26fb5 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3d93 expref: -99 pid: 66517 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419417:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206e800/0xe83a7ce271e26f0d lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3ccf expref: -99 pid: 66510 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419423:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746400/0xe83a7ce271e26e96 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3c0b expref: -99 pid: 66500 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419449:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206fa00/0xe83a7ce271e26db6 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3b32 expref: -99 pid: 66489 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419458:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d22c00/0xe83a7ce271e26d0e lrc: 0/0,0 mode: --/PR res: [0x240000403:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3a6e expref: -99 pid: 66482 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419469:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23c00/0xe83a7ce271e26c97 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe39aa expref: -99 pid: 66473 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419479:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23a00/0xe83a7ce271e26bef lrc: 0/0,0 mode: --/PR res: [0x240000403:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe38e6 expref: -99 pid: 66465 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419531:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f72e00/0xe83a7ce271e26bb0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3837 expref: -99 pid: 66460 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419544:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f73600/0xe83a7ce271e26b71 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3788 expref: -99 pid: 66455 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419556:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83f73a00/0xe83a7ce271e26b32 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe36d9 expref: -99 pid: 66450 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419595:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d22200/0xe83a7ce271e26af3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe362a expref: -99 pid: 66444 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419605:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476800/0xe83a7ce271e26a7c lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3566 expref: -99 pid: 66437 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419614:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476600/0xe83a7ce271e26a3d lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe34b7 expref: -99 pid: 66432 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419623:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81476c00/0xe83a7ce271e269f7 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe33f3 expref: -99 pid: 66427 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419633:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d22e00/0xe83a7ce271e269b8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3344 expref: -99 pid: 66421 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419657:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23e00/0xe83a7ce271e26941 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe3280 expref: -99 pid: 66414 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419663:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23800/0xe83a7ce271e268ca lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe31bc expref: -99 pid: 66405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419670:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206a000/0xe83a7ce271e267ea lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe30e3 expref: -99 pid: 66393 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419676:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23200/0xe83a7ce271e2670a lrc: 0/0,0 mode: --/PR res: [0x240000403:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe300a expref: -99 pid: 66381 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419684:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9206b200/0xe83a7ce271e26662 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2f46 expref: -99 pid: 66374 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419690:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23000/0xe83a7ce271e2661c lrc: 0/0,0 mode: --/PR res: [0x240000403:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2e82 expref: -99 pid: 66369 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419697:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9230b400/0xe83a7ce271e265dd lrc: 0/0,0 mode: --/PR res: [0x240000403:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2dd3 expref: -99 pid: 66364 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419703:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2000/0xe83a7ce271e2659e lrc: 0/0,0 mode: --/PR res: [0x240000403:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2d24 expref: -99 pid: 66358 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419709:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8625ce00/0xe83a7ce271e2655f lrc: 0/0,0 mode: --/PR res: [0x240000403:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2c75 expref: -99 pid: 66353 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419715:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b855c3200/0xe83a7ce271e264e8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2bb1 expref: -99 pid: 66344 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419722:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8625de00/0xe83a7ce271e26439 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2ad8 expref: -99 pid: 66337 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419728:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2600/0xe83a7ce271e263f3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2a14 expref: -99 pid: 66331 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419735:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81477200/0xe83a7ce271e2637c lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2950 expref: -99 pid: 66322 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419742:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee3000/0xe83a7ce271e2629c lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe2877 expref: -99 pid: 66313 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419750:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee2400/0xe83a7ce271e26256 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe27b3 expref: -99 pid: 66308 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419757:0:6177:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85ee3400/0xe83a7ce271e261df lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe26ef expref: -99 pid: 66298 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075509.419764:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075509.419765:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075509.419767:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075509.736564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075509.736568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075509.736622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075509.736629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:3.0:1769075510.440595:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075510.760537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075510.760553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075510.760553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075511.464654:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075511.784650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075511.784650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075511.784651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075512.488693:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075512.808622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075512.808635:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075512.808654:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075513.512675:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075513.832554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075513.832554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075513.832569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075513.832587:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075513.832592:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075513.832596:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075513.832605:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2499 00000100:00080000:1.0:1769075513.832615:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075513.832621:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075513.832667:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616396928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075513.832716:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075513.832720:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075513.832721:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075513.832725:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2499 00000100:00080000:1.0:1769075513.832728:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075513.832731:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075513.832739:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075513.832753:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616397056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075513.832778:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075513.832781:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075513.832782:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075513.832785:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2499 00000100:00080000:1.0:1769075513.832788:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075513.832791:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075513.832795:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075513.832804:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616397184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075513.832811:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075513.832813:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075513.832815:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075513.832817:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2499 00000100:00080000:1.0:1769075513.832822:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075513.832825:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075513.832829:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075513.832852:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4e00 x1855007616397312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075513.832875:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075513.832876:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075513.832877:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075513.832880:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2499 00000100:00080000:1.0:1769075513.832882:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075513.832884:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075513.832886:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075513.832892:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616397440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075513.832898:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075513.832901:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075513.832902:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075513.832905:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2499 00000100:00080000:1.0:1769075513.832907:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075513.832910:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075513.832913:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075513.832918:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616397568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075513.832924:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075513.832926:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075513.832927:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075513.832941:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2499 00000100:00080000:1.0:1769075513.832944:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075513.832947:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075513.832951:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075513.832961:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616397696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075514.536621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075514.856604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075514.856606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075514.856606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075515.560643:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075515.880628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075515.880702:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075515.880747:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075516.584679:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075516.904607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075516.904677:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075516.904692:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075517.608596:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075517.928579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075517.928595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:3.0:1769075518.632574:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075518.952567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075518.952631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075518.952640:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075518.952649:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769075518.952652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075518.952654:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075518.952669:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2504 00000100:00080000:1.0:1769075518.952681:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075518.952687:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075518.952721:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616397824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075518.952746:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075518.952749:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075518.952750:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075518.952753:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2504 00000100:00080000:1.0:1769075518.952755:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075518.952769:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075518.952776:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075518.952785:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616397952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075518.952792:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075518.952795:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075518.952796:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075518.952799:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2504 00000100:00080000:1.0:1769075518.952801:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075518.952804:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075518.952807:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075518.952818:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616398080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075518.952840:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075518.952855:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075518.952858:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075518.952863:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2504 00000100:00080000:1.0:1769075518.952867:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075518.952873:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075518.952880:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075518.952901:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616398208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075518.952913:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075518.952918:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075518.952921:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075518.952924:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2504 00000100:00080000:1.0:1769075518.952928:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075518.952931:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075518.952937:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075518.952946:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7b80 x1855007616398336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075518.952963:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075518.952965:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075518.952966:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075518.952970:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2504 00000100:00080000:1.0:1769075518.952972:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075518.952981:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075518.952992:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075518.953005:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6680 x1855007616398464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075518.953018:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075518.953021:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075518.953023:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075518.953027:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2504 00000100:00080000:1.0:1769075518.953029:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075518.953031:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075518.953035:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075518.953054:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5180 x1855007616398592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075519.656783:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 67/67 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1769075519.656848:0:68227:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 67 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1769075519.656854:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075519.656866:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5c00/0xe83a7ce271e29888 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x154:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe866b expref: -99 pid: 67195 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.656901:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x154:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.656944:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2c00/0xe83a7ce271e29811 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x153:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe85a7 expref: -99 pid: 67187 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.656955:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x153:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.656970:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b6c00/0xe83a7ce271e297d2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x152:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe84f8 expref: -99 pid: 67182 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.656980:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x152:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.656992:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b6800/0xe83a7ce271e2978c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x151:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8434 expref: -99 pid: 67177 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657012:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x151:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657023:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744200/0xe83a7ce271e2974d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x150:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8385 expref: -99 pid: 67172 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657033:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x150:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657045:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3400/0xe83a7ce271e2970e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x14f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe82d6 expref: -99 pid: 67166 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657055:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x14f:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657068:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745600/0xe83a7ce271e296cf lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x14e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8227 expref: -99 pid: 67161 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657077:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x14e:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657086:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b7e00/0xe83a7ce271e29690 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x14d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8178 expref: -99 pid: 67156 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657092:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x14d:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657100:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2000/0xe83a7ce271e29651 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x14c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe80c9 expref: -99 pid: 67151 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657106:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x14c:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657113:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2800/0xe83a7ce271e295da lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x14b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8005 expref: -99 pid: 67141 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657118:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x14b:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657125:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b7000/0xe83a7ce271e2952b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x14a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe7f2c expref: -99 pid: 67132 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657154:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x14a:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657165:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8e00/0xe83a7ce271e2944b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x149:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe7e53 expref: -99 pid: 67121 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657172:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x149:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657179:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0200/0xe83a7ce271e2936b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x148:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe7d7a expref: -99 pid: 67110 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657185:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x148:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657193:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0400/0xe83a7ce271e292c3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x147:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe7cb6 expref: -99 pid: 67102 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657198:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x147:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657205:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3e00/0xe83a7ce271e29284 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x146:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe7c07 expref: -99 pid: 67097 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657211:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x146:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657218:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2e00/0xe83a7ce271e29245 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x145:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe7b58 expref: -99 pid: 67092 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657225:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x145:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657232:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9400/0xe83a7ce271e29206 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x144:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe7aa9 expref: -99 pid: 67087 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657238:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x144:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657247:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1000/0xe83a7ce271e291c7 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x143:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe79fa expref: -99 pid: 67081 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657257:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x143:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657270:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9e00/0xe83a7ce271e29188 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x142:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe794b expref: -99 pid: 67076 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657281:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x142:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657292:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3000/0xe83a7ce271e29149 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x141:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe789c expref: -99 pid: 67071 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657302:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x141:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657314:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0e00/0xe83a7ce271e2910a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x140:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe77ed expref: -99 pid: 67066 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657324:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x140:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657338:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8600/0xe83a7ce271e29093 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x13f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe7729 expref: -99 pid: 67056 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657349:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x13f:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657361:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1c00/0xe83a7ce271e28fb3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x13e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe7650 expref: -99 pid: 67045 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657372:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x13e:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657385:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa8400/0xe83a7ce271e28f3c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x13d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe758c expref: -99 pid: 67040 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657395:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x13d:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657407:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1e00/0xe83a7ce271e28efd lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x13c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe74dd expref: -99 pid: 67035 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657418:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x13c:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657431:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3800/0xe83a7ce271e28ebe lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x13b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe742e expref: -99 pid: 67029 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657442:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x13b:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657503:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa9000/0xe83a7ce271e28e7f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x13a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe737f expref: -99 pid: 67023 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657513:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x13a:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657523:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0c200/0xe83a7ce271e28e08 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x139:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe72bb expref: -99 pid: 67014 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657529:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x139:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657541:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519600/0xe83a7ce271e28d28 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x138:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe71e2 expref: -99 pid: 67003 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657549:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x138:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657560:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9200/0xe83a7ce271e28c80 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x137:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe711e expref: -99 pid: 66995 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657566:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x137:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657576:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551be00/0xe83a7ce271e28c41 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x136:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe706f expref: -99 pid: 66990 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657586:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x136:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657615:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8800/0xe83a7ce271e28c02 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x135:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6fc0 expref: -99 pid: 66985 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657626:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x135:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657640:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746800/0xe83a7ce271e28bc3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x134:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6f11 expref: -99 pid: 66980 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657651:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x134:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657665:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746e00/0xe83a7ce271e28b84 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x133:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6e62 expref: -99 pid: 66974 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657675:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x133:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657687:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5000/0xe83a7ce271e28b0d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x132:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6d9e expref: -99 pid: 66965 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657698:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x132:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657711:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621b400/0xe83a7ce271e28a65 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x131:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6cda expref: -99 pid: 66958 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657747:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x131:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657761:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322600/0xe83a7ce271e289ee lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x130:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6c16 expref: -99 pid: 66951 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657771:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x130:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657783:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203400/0xe83a7ce271e28977 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6b52 expref: -99 pid: 66941 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657792:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x12f:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657804:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551a000/0xe83a7ce271e288c8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6a79 expref: -99 pid: 66934 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657811:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x12e:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657818:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5200/0xe83a7ce271e28889 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe69ca expref: -99 pid: 66929 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657824:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x12d:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657831:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4e00/0xe83a7ce271e2884a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe691b expref: -99 pid: 66924 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657837:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x12c:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657844:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ff800/0xe83a7ce271e2880b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe686c expref: -99 pid: 66918 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657849:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x12b:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657856:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519000/0xe83a7ce271e287cc lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe67bd expref: -99 pid: 66913 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657862:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x12a:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657869:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519400/0xe83a7ce271e28755 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x129:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe66f9 expref: -99 pid: 66904 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657874:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x129:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657881:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561600/0xe83a7ce271e286ad lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x128:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6635 expref: -99 pid: 66897 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657885:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x128:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657892:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5800/0xe83a7ce271e28667 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x127:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6571 expref: -99 pid: 66891 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657897:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x127:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657904:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560200/0xe83a7ce271e285f0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x126:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe64ad expref: -99 pid: 66882 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657909:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x126:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657916:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4c00/0xe83a7ce271e28541 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x125:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe63d4 expref: -99 pid: 66875 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657921:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x125:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657928:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4a00/0xe83a7ce271e284ca lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x124:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6310 expref: -99 pid: 66866 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657933:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x124:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657940:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560800/0xe83a7ce271e283ea lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x123:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6237 expref: -99 pid: 66854 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657946:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x123:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657953:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519800/0xe83a7ce271e28342 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x122:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6173 expref: -99 pid: 66847 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657960:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x122:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657967:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113a800/0xe83a7ce271e28303 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x121:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe60c4 expref: -99 pid: 66842 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.657990:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x121:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.657998:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4400/0xe83a7ce271e282c4 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x120:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe6015 expref: -99 pid: 66837 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658005:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x120:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658012:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113b200/0xe83a7ce271e28285 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5f66 expref: -99 pid: 66831 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658018:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x11f:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658025:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0c400/0xe83a7ce271e28246 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5eb7 expref: -99 pid: 66826 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658030:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x11e:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658037:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0ca00/0xe83a7ce271e281cf lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5df3 expref: -99 pid: 66817 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658043:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x11d:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658050:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f50200/0xe83a7ce271e28127 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5d2f expref: -99 pid: 66810 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658055:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x11c:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658062:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0d200/0xe83a7ce271e280e8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5c80 expref: -99 pid: 66804 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658068:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x11b:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658075:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113ba00/0xe83a7ce271e280a9 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5bd1 expref: -99 pid: 66799 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658081:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x11a:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658088:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113ae00/0xe83a7ce271e2806a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x119:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5b22 expref: -99 pid: 66794 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658094:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x119:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658102:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113a200/0xe83a7ce271e2802b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x118:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5a73 expref: -99 pid: 66789 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658107:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x118:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658115:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f50a00/0xe83a7ce271e27fec lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x117:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe59c4 expref: -99 pid: 66783 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658121:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x117:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658128:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0d600/0xe83a7ce271e27f75 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x116:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5900 expref: -99 pid: 66774 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658170:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x116:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658179:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561200/0xe83a7ce271e27ec6 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x115:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5827 expref: -99 pid: 66767 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658185:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x115:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658192:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23400/0xe83a7ce271e27e4f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe5763 expref: -99 pid: 66758 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658198:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x114:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658205:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113bc00/0xe83a7ce271e27d6f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x113:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe568a expref: -99 pid: 66746 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658210:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x113:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658218:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85518400/0xe83a7ce271e27cc7 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x112:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe55c6 expref: -99 pid: 66739 timeout: 0 lvb_type: 3 00000080:00200000:1.0:1769075519.658224:0:68227:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x112:0x0]: losing layout lock 00010000:00010000:1.0:1769075519.658263:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5c00/0xe83a7ce271e29888 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x154:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe866b expref: -99 pid: 67195 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658272:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2c00/0xe83a7ce271e29811 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x153:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe85a7 expref: -99 pid: 67187 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658279:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b6c00/0xe83a7ce271e297d2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x152:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe84f8 expref: -99 pid: 67182 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658286:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b6800/0xe83a7ce271e2978c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x151:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8434 expref: -99 pid: 67177 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658294:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744200/0xe83a7ce271e2974d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x150:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8385 expref: -99 pid: 67172 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658302:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3400/0xe83a7ce271e2970e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x14f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe82d6 expref: -99 pid: 67166 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658309:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745600/0xe83a7ce271e296cf lrc: 1/0,0 mode: --/PR res: [0x240000403:0x14e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8227 expref: -99 pid: 67161 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658315:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b7e00/0xe83a7ce271e29690 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x14d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8178 expref: -99 pid: 67156 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658321:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2000/0xe83a7ce271e29651 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x14c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe80c9 expref: -99 pid: 67151 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658325:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2800/0xe83a7ce271e295da lrc: 1/0,0 mode: --/PR res: [0x240000403:0x14b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8005 expref: -99 pid: 67141 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658333:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b7000/0xe83a7ce271e2952b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x14a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7f2c expref: -99 pid: 67132 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658338:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8e00/0xe83a7ce271e2944b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x149:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7e53 expref: -99 pid: 67121 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658342:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0200/0xe83a7ce271e2936b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x148:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7d7a expref: -99 pid: 67110 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658346:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0400/0xe83a7ce271e292c3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x147:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7cb6 expref: -99 pid: 67102 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658350:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3e00/0xe83a7ce271e29284 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x146:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7c07 expref: -99 pid: 67097 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658354:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2e00/0xe83a7ce271e29245 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x145:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7b58 expref: -99 pid: 67092 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658357:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9400/0xe83a7ce271e29206 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x144:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7aa9 expref: -99 pid: 67087 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658361:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1000/0xe83a7ce271e291c7 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x143:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe79fa expref: -99 pid: 67081 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658365:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9e00/0xe83a7ce271e29188 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x142:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe794b expref: -99 pid: 67076 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658369:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3000/0xe83a7ce271e29149 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x141:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe789c expref: -99 pid: 67071 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658377:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0e00/0xe83a7ce271e2910a lrc: 1/0,0 mode: --/PR res: [0x240000403:0x140:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe77ed expref: -99 pid: 67066 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658381:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8600/0xe83a7ce271e29093 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x13f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7729 expref: -99 pid: 67056 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658385:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1c00/0xe83a7ce271e28fb3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x13e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7650 expref: -99 pid: 67045 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658389:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa8400/0xe83a7ce271e28f3c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x13d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe758c expref: -99 pid: 67040 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658393:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1e00/0xe83a7ce271e28efd lrc: 1/0,0 mode: --/PR res: [0x240000403:0x13c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe74dd expref: -99 pid: 67035 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658397:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3800/0xe83a7ce271e28ebe lrc: 1/0,0 mode: --/PR res: [0x240000403:0x13b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe742e expref: -99 pid: 67029 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658401:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa9000/0xe83a7ce271e28e7f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x13a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe737f expref: -99 pid: 67023 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658405:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0c200/0xe83a7ce271e28e08 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x139:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe72bb expref: -99 pid: 67014 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658409:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519600/0xe83a7ce271e28d28 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x138:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe71e2 expref: -99 pid: 67003 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658413:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9200/0xe83a7ce271e28c80 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x137:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe711e expref: -99 pid: 66995 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658420:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551be00/0xe83a7ce271e28c41 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x136:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe706f expref: -99 pid: 66990 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658424:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8800/0xe83a7ce271e28c02 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x135:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6fc0 expref: -99 pid: 66985 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658428:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746800/0xe83a7ce271e28bc3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x134:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6f11 expref: -99 pid: 66980 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658432:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746e00/0xe83a7ce271e28b84 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x133:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6e62 expref: -99 pid: 66974 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658439:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5000/0xe83a7ce271e28b0d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x132:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6d9e expref: -99 pid: 66965 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658445:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621b400/0xe83a7ce271e28a65 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x131:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6cda expref: -99 pid: 66958 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658451:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322600/0xe83a7ce271e289ee lrc: 1/0,0 mode: --/PR res: [0x240000403:0x130:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6c16 expref: -99 pid: 66951 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658458:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203400/0xe83a7ce271e28977 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x12f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6b52 expref: -99 pid: 66941 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658465:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551a000/0xe83a7ce271e288c8 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x12e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6a79 expref: -99 pid: 66934 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658495:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5200/0xe83a7ce271e28889 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x12d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe69ca expref: -99 pid: 66929 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658508:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4e00/0xe83a7ce271e2884a lrc: 1/0,0 mode: --/PR res: [0x240000403:0x12c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe691b expref: -99 pid: 66924 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658515:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ff800/0xe83a7ce271e2880b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x12b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe686c expref: -99 pid: 66918 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658523:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519000/0xe83a7ce271e287cc lrc: 1/0,0 mode: --/PR res: [0x240000403:0x12a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe67bd expref: -99 pid: 66913 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658530:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519400/0xe83a7ce271e28755 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x129:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe66f9 expref: -99 pid: 66904 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658537:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561600/0xe83a7ce271e286ad lrc: 1/0,0 mode: --/PR res: [0x240000403:0x128:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6635 expref: -99 pid: 66897 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658544:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5800/0xe83a7ce271e28667 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x127:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6571 expref: -99 pid: 66891 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658552:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560200/0xe83a7ce271e285f0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x126:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe64ad expref: -99 pid: 66882 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658560:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4c00/0xe83a7ce271e28541 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x125:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe63d4 expref: -99 pid: 66875 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658575:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4a00/0xe83a7ce271e284ca lrc: 1/0,0 mode: --/PR res: [0x240000403:0x124:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6310 expref: -99 pid: 66866 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658582:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560800/0xe83a7ce271e283ea lrc: 1/0,0 mode: --/PR res: [0x240000403:0x123:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6237 expref: -99 pid: 66854 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658594:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519800/0xe83a7ce271e28342 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x122:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6173 expref: -99 pid: 66847 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658600:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113a800/0xe83a7ce271e28303 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x121:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe60c4 expref: -99 pid: 66842 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658604:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4400/0xe83a7ce271e282c4 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x120:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6015 expref: -99 pid: 66837 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658608:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113b200/0xe83a7ce271e28285 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x11f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5f66 expref: -99 pid: 66831 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658612:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0c400/0xe83a7ce271e28246 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x11e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5eb7 expref: -99 pid: 66826 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658616:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0ca00/0xe83a7ce271e281cf lrc: 1/0,0 mode: --/PR res: [0x240000403:0x11d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5df3 expref: -99 pid: 66817 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658620:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f50200/0xe83a7ce271e28127 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x11c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5d2f expref: -99 pid: 66810 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658624:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0d200/0xe83a7ce271e280e8 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x11b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5c80 expref: -99 pid: 66804 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658628:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113ba00/0xe83a7ce271e280a9 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x11a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5bd1 expref: -99 pid: 66799 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658632:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113ae00/0xe83a7ce271e2806a lrc: 1/0,0 mode: --/PR res: [0x240000403:0x119:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5b22 expref: -99 pid: 66794 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658639:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113a200/0xe83a7ce271e2802b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x118:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5a73 expref: -99 pid: 66789 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658644:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f50a00/0xe83a7ce271e27fec lrc: 1/0,0 mode: --/PR res: [0x240000403:0x117:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe59c4 expref: -99 pid: 66783 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658648:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0d600/0xe83a7ce271e27f75 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x116:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5900 expref: -99 pid: 66774 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658651:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561200/0xe83a7ce271e27ec6 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x115:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5827 expref: -99 pid: 66767 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658655:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23400/0xe83a7ce271e27e4f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5763 expref: -99 pid: 66758 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658659:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113bc00/0xe83a7ce271e27d6f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x113:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe568a expref: -99 pid: 66746 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658663:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85518400/0xe83a7ce271e27cc7 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x112:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe55c6 expref: -99 pid: 66739 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658667:0:68227:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 67 locks packed 00010000:00010000:1.0:1769075519.658695:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5c00/0xe83a7ce271e29888 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x154:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe866b expref: -99 pid: 67195 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658713:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2c00/0xe83a7ce271e29811 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x153:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe85a7 expref: -99 pid: 67187 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658723:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b6c00/0xe83a7ce271e297d2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x152:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe84f8 expref: -99 pid: 67182 timeout: 0 lvb_type: 3 00000100:00080000:0.0:1769075519.658736:0:2418:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b868c5180 x1855007616398720/t0(0) o103->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 848/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1769075519.658736:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b6800/0xe83a7ce271e2978c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x151:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8434 expref: -99 pid: 67177 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658755:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744200/0xe83a7ce271e2974d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x150:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8385 expref: -99 pid: 67172 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658766:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3400/0xe83a7ce271e2970e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x14f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe82d6 expref: -99 pid: 67166 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658776:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745600/0xe83a7ce271e296cf lrc: 0/0,0 mode: --/PR res: [0x240000403:0x14e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8227 expref: -99 pid: 67161 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658788:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b7e00/0xe83a7ce271e29690 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x14d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8178 expref: -99 pid: 67156 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658796:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2000/0xe83a7ce271e29651 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x14c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe80c9 expref: -99 pid: 67151 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658804:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2800/0xe83a7ce271e295da lrc: 0/0,0 mode: --/PR res: [0x240000403:0x14b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8005 expref: -99 pid: 67141 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658812:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b7000/0xe83a7ce271e2952b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x14a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7f2c expref: -99 pid: 67132 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658819:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8e00/0xe83a7ce271e2944b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x149:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7e53 expref: -99 pid: 67121 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658827:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0200/0xe83a7ce271e2936b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x148:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7d7a expref: -99 pid: 67110 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658852:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0400/0xe83a7ce271e292c3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x147:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7cb6 expref: -99 pid: 67102 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658860:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3e00/0xe83a7ce271e29284 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x146:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7c07 expref: -99 pid: 67097 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658867:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2e00/0xe83a7ce271e29245 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x145:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7b58 expref: -99 pid: 67092 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658873:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9400/0xe83a7ce271e29206 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x144:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7aa9 expref: -99 pid: 67087 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658881:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1000/0xe83a7ce271e291c7 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x143:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe79fa expref: -99 pid: 67081 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658889:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9e00/0xe83a7ce271e29188 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x142:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe794b expref: -99 pid: 67076 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658896:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3000/0xe83a7ce271e29149 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x141:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe789c expref: -99 pid: 67071 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658903:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0e00/0xe83a7ce271e2910a lrc: 0/0,0 mode: --/PR res: [0x240000403:0x140:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe77ed expref: -99 pid: 67066 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658912:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8600/0xe83a7ce271e29093 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x13f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7729 expref: -99 pid: 67056 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658919:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1c00/0xe83a7ce271e28fb3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x13e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe7650 expref: -99 pid: 67045 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658926:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa8400/0xe83a7ce271e28f3c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x13d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe758c expref: -99 pid: 67040 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658936:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1e00/0xe83a7ce271e28efd lrc: 0/0,0 mode: --/PR res: [0x240000403:0x13c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe74dd expref: -99 pid: 67035 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658948:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f3800/0xe83a7ce271e28ebe lrc: 0/0,0 mode: --/PR res: [0x240000403:0x13b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe742e expref: -99 pid: 67029 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658960:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa9000/0xe83a7ce271e28e7f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x13a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe737f expref: -99 pid: 67023 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658982:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0c200/0xe83a7ce271e28e08 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x139:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe72bb expref: -99 pid: 67014 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.658995:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519600/0xe83a7ce271e28d28 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x138:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe71e2 expref: -99 pid: 67003 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659007:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9200/0xe83a7ce271e28c80 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x137:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe711e expref: -99 pid: 66995 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659020:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551be00/0xe83a7ce271e28c41 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x136:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe706f expref: -99 pid: 66990 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659032:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8800/0xe83a7ce271e28c02 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x135:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6fc0 expref: -99 pid: 66985 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659044:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746800/0xe83a7ce271e28bc3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x134:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6f11 expref: -99 pid: 66980 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659056:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b84746e00/0xe83a7ce271e28b84 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x133:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6e62 expref: -99 pid: 66974 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659064:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5000/0xe83a7ce271e28b0d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x132:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6d9e expref: -99 pid: 66965 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659072:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8621b400/0xe83a7ce271e28a65 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x131:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6cda expref: -99 pid: 66958 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659079:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b81322600/0xe83a7ce271e289ee lrc: 0/0,0 mode: --/PR res: [0x240000403:0x130:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6c16 expref: -99 pid: 66951 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659086:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85203400/0xe83a7ce271e28977 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x12f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6b52 expref: -99 pid: 66941 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659094:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551a000/0xe83a7ce271e288c8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x12e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6a79 expref: -99 pid: 66934 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659105:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5200/0xe83a7ce271e28889 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x12d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe69ca expref: -99 pid: 66929 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659115:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4e00/0xe83a7ce271e2884a lrc: 0/0,0 mode: --/PR res: [0x240000403:0x12c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe691b expref: -99 pid: 66924 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659126:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b852ff800/0xe83a7ce271e2880b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x12b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe686c expref: -99 pid: 66918 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659153:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519000/0xe83a7ce271e287cc lrc: 0/0,0 mode: --/PR res: [0x240000403:0x12a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe67bd expref: -99 pid: 66913 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659166:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519400/0xe83a7ce271e28755 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x129:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe66f9 expref: -99 pid: 66904 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659193:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561600/0xe83a7ce271e286ad lrc: 0/0,0 mode: --/PR res: [0x240000403:0x128:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6635 expref: -99 pid: 66897 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659203:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa5800/0xe83a7ce271e28667 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x127:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6571 expref: -99 pid: 66891 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659213:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560200/0xe83a7ce271e285f0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x126:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe64ad expref: -99 pid: 66882 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659221:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4c00/0xe83a7ce271e28541 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x125:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe63d4 expref: -99 pid: 66875 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659228:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4a00/0xe83a7ce271e284ca lrc: 0/0,0 mode: --/PR res: [0x240000403:0x124:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6310 expref: -99 pid: 66866 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659234:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90560800/0xe83a7ce271e283ea lrc: 0/0,0 mode: --/PR res: [0x240000403:0x123:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6237 expref: -99 pid: 66854 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659241:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85519800/0xe83a7ce271e28342 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x122:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6173 expref: -99 pid: 66847 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659252:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113a800/0xe83a7ce271e28303 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x121:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe60c4 expref: -99 pid: 66842 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659264:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83fa4400/0xe83a7ce271e282c4 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x120:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe6015 expref: -99 pid: 66837 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659278:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113b200/0xe83a7ce271e28285 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x11f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5f66 expref: -99 pid: 66831 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659289:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0c400/0xe83a7ce271e28246 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x11e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5eb7 expref: -99 pid: 66826 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659300:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0ca00/0xe83a7ce271e281cf lrc: 0/0,0 mode: --/PR res: [0x240000403:0x11d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5df3 expref: -99 pid: 66817 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659311:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f50200/0xe83a7ce271e28127 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x11c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5d2f expref: -99 pid: 66810 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659322:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0d200/0xe83a7ce271e280e8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x11b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5c80 expref: -99 pid: 66804 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659333:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113ba00/0xe83a7ce271e280a9 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x11a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5bd1 expref: -99 pid: 66799 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659343:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113ae00/0xe83a7ce271e2806a lrc: 0/0,0 mode: --/PR res: [0x240000403:0x119:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5b22 expref: -99 pid: 66794 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659354:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113a200/0xe83a7ce271e2802b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x118:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5a73 expref: -99 pid: 66789 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659364:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f50a00/0xe83a7ce271e27fec lrc: 0/0,0 mode: --/PR res: [0x240000403:0x117:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe59c4 expref: -99 pid: 66783 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659372:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0d600/0xe83a7ce271e27f75 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x116:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5900 expref: -99 pid: 66774 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659379:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561200/0xe83a7ce271e27ec6 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x115:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5827 expref: -99 pid: 66767 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659386:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d23400/0xe83a7ce271e27e4f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe5763 expref: -99 pid: 66758 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659393:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9113bc00/0xe83a7ce271e27d6f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x113:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe568a expref: -99 pid: 66746 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659400:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85518400/0xe83a7ce271e27cc7 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x112:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe55c6 expref: -99 pid: 66739 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075519.659407:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075519.659408:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075519.659410:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075519.976690:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075519.976746:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075519.976762:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075519.976794:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075520.680699:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075521.000605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075521.000681:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075521.000682:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075521.704584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075522.024670:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075522.024674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075522.024688:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075522.728805:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075523.048617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075523.048629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075523.048632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075523.048729:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075523.752580:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075524.072554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075524.072575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075524.072582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075524.072600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075524.072700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616398848:192.168.204.132@tcp:250:kworker.0 00000800:00000100:0.0:1769075524.072797:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075524.072853:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075524/real 1769075524] req@ffff995b85ac9880 x1855007616398848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075524.072903:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616398848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075524.072925:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075524.072934:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075524.072941:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616398848:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075524.072971:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616398976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075524.072999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075524.073014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075524/real 1769075524] req@ffff995b85ac9500 x1855007616398976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075524.073025:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9500 x1855007616398976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075524.073032:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075524.073034:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075524.073037:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616398976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075524.073042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616399104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075524.073076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075524.073088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075524/real 1769075524] req@ffff995b85acad80 x1855007616399104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075524.073102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acad80 x1855007616399104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075524.073118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075524.073124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075524.073130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616399104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075524.073140:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616399232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075524.073170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075524.073185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075524/real 1769075524] req@ffff995b85acb800 x1855007616399232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075524.073213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616399232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075524.073221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075524.073224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075524.073228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616399232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075524.073234:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616399360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075524.073253:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075524.073265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075524/real 1769075524] req@ffff995b85ac8a80 x1855007616399360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075524.073273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616399360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075524.073280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075524.073282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075524.073284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616399360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075524.073288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616399488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075524.073297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075524.073303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075524/real 1769075524] req@ffff995b85acbb80 x1855007616399488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075524.073310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616399488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075524.073316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075524.073318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075524.073320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616399488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075524.073323:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616399616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075524.073332:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075524.073338:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075524/real 1769075524] req@ffff995b85aca300 x1855007616399616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075524.073346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616399616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075524.073360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075524.073362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075524.073365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616399616:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769075524.776619:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075525.096610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075525.096677:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075525.096694:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:3.0:1769075525.800581:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075526.120537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075526.120554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075526.120582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075526.824732:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075527.144527:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075527.144552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075527.144552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075527.848683:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075528.168560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075528.168588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075528.169535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075528.872757:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075529.192598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075529.192656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075529.192665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075529.192938:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616399744:192.168.204.132@tcp:250:kworker.0 00000800:00000100:0.0:1769075529.193045:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075529.193104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075529/real 1769075529] req@ffff995b85aca680 x1855007616399744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075529.193145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca680 x1855007616399744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075529.193171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075529.193182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075529.193193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616399744:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075529.193229:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616399872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075529.193261:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075529.193278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075529/real 1769075529] req@ffff995b85ac9c00 x1855007616399872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075529.193311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616399872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075529.193333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075529.193346:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075529.193355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616399872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075529.193364:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616400000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075529.193393:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075529.193409:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075529/real 1769075529] req@ffff995b85ac8380 x1855007616400000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075529.193431:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616400000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075529.193486:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075529.193492:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075529.193497:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616400000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075529.193526:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616400128:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075529.193558:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075529.193580:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075529/real 1769075529] req@ffff995b85acb100 x1855007616400128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075529.193612:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616400128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075529.193657:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075529.193670:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075529.193687:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616400128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075529.193711:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616400256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075529.193754:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075529.193776:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075529/real 1769075529] req@ffff995b85acaa00 x1855007616400256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075529.193822:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616400256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075529.193833:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075529.193837:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075529.193844:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616400256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075529.193861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616400384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075529.193879:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075529.193889:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075529/real 1769075529] req@ffff995b85ac8380 x1855007616400384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075529.193902:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616400384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075529.193910:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075529.193913:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075529.193916:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616400384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075529.193922:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616400512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075529.193937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075529.193947:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075529/real 1769075529] req@ffff995b85ac9c00 x1855007616400512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075529.193960:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616400512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075529.193997:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075529.194003:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075529.194009:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616400512:192.168.204.132@tcp:38:kworker.0 00010000:00010000:3.0:1769075529.896597:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 16/16 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1769075529.896672:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 71/87 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1769075529.896697:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 16/103 locks to regular blp list, 3 blwis in pool 00010000:00010000:0.0:1769075529.896727:0:13582:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 16 locks of 87 total in blp. (2 blwis in pool) 00010000:00010000:3.0:1769075529.896736:0:13603:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 71 locks of 16 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1769075529.896737:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621ae00/0xe83a7ce271e2613e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe262b expref: -99 pid: 66291 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.896749:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83544000/0xe83a7ce271e2b6fe lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x19b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98febae4 expref: -99 pid: 67690 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075529.896764:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1769075529.896769:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xd2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.896785:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x19b:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.896830:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xd2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075529.896862:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b922f8c00/0xe83a7ce271e2b6bf lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x19a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feba35 expref: -99 pid: 67685 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.896876:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206bc00/0xe83a7ce271e2656d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe2c8a expref: -99 pid: 66353 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075529.896896:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xda:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.896897:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x19a:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.896936:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xda:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075529.896942:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f6c00/0xe83a7ce271e2b680 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x199:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb986 expref: -99 pid: 67680 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.896993:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x199:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.897002:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621b600/0xe83a7ce271e26264 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe27c8 expref: -99 pid: 66308 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.897012:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83545000/0xe83a7ce271e2b641 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x198:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb8d7 expref: -99 pid: 67675 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897052:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xd4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.897072:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x198:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897080:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xd4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075529.897096:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8625d600/0xe83a7ce271e26447 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe2aed expref: -99 pid: 66337 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.897100:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b3e00/0xe83a7ce271e2b5ca lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x197:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb813 expref: -99 pid: 67665 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897125:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xd8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.897132:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x197:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897152:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xd8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075529.897154:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9234fc00/0xe83a7ce271e2b4ea lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x196:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb73a expref: -99 pid: 67654 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.897165:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee2a00/0xe83a7ce271e2638a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe2965 expref: -99 pid: 66322 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.897166:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x196:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897181:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xd6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1769075529.897186:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f7400/0xe83a7ce271e2b43b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x195:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb661 expref: -99 pid: 67647 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897205:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xd6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:3.0:1769075529.897207:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x195:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.897222:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621b200/0xe83a7ce271e25f69 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe2330 expref: -99 pid: 66262 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.897226:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834faa00/0xe83a7ce271e2b3fc lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x194:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb5b2 expref: -99 pid: 67642 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897233:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xce:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.897237:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x194:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.897248:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d6aa00/0xe83a7ce271e2b3bd lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x193:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb503 expref: -99 pid: 67636 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.897257:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x193:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897267:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xce:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075529.897275:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fa200/0xe83a7ce271e2b377 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x192:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb43f expref: -99 pid: 67631 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.897280:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621bc00/0xe83a7ce271e25eb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x61:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe21bd expref: -99 pid: 66250 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.897285:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x192:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897297:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xcc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1769075529.897301:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d6b400/0xe83a7ce271e2b300 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x191:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb37b expref: -99 pid: 67622 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.897312:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x191:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897322:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xcc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075529.897328:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb200/0xe83a7ce271e2b220 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x190:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb2a2 expref: -99 pid: 67611 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.897335:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f07600/0xe83a7ce271e25dfd lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x60:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe204a expref: -99 pid: 66235 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.897339:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x190:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897350:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xca:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1769075529.897355:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fa000/0xe83a7ce271e2b178 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb1de expref: -99 pid: 67603 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.897367:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x18f:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897372:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xca:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075529.897376:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb800/0xe83a7ce271e2b139 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb12f expref: -99 pid: 67598 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.897386:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f77000/0xe83a7ce271e25cd7 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x5f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe1ead expref: -99 pid: 66221 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.897402:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x18e:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.897422:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fae00/0xe83a7ce271e2b0fa lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feb080 expref: -99 pid: 67593 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897429:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xc8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.897442:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x18d:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.897457:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb000/0xe83a7ce271e2b0bb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feafd1 expref: -99 pid: 67588 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897477:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xc8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075529.897490:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9230a400/0xe83a7ce271e25c59 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x5e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe1d4f expref: -99 pid: 66210 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.897493:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x18c:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897508:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xc6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1769075529.897514:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fba00/0xe83a7ce271e2b07c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feaf22 expref: -99 pid: 67582 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897533:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xc6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:3.0:1769075529.897539:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x18b:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.897545:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81477e00/0xe83a7ce271e25b3a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x5d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe1bc7 expref: -99 pid: 66194 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.897556:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83545e00/0xe83a7ce271e2b005 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feae5e expref: -99 pid: 67573 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897563:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xc4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.897567:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x18a:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.897587:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fbc00/0xe83a7ce271e2af5d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x189:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fead9a expref: -99 pid: 67566 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897588:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xc4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:3.0:1769075529.897607:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x189:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.897609:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83547a00/0xe83a7ce271e25a14 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x5c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe1a2a expref: -99 pid: 66177 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075529.897623:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xc2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1769075529.897624:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744600/0xe83a7ce271e2af1e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x188:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feaceb expref: -99 pid: 67561 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897641:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xc2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:3.0:1769075529.897643:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x188:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.897657:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83547e00/0xe83a7ce271e2598f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x5b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe18b7 expref: -99 pid: 66167 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.897663:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b7c00/0xe83a7ce271e2aea7 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x187:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feac27 expref: -99 pid: 67551 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897678:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xc0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.897679:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x187:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.897699:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0da00/0xe83a7ce271e2adff lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x186:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feab63 expref: -99 pid: 67544 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897705:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xc0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:3.0:1769075529.897708:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x186:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.897726:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83546800/0xe83a7ce271e258d9 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x5a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe1744 expref: -99 pid: 66152 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.897730:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b80030c00/0xe83a7ce271e2adc0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x185:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98feaab4 expref: -99 pid: 67539 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897759:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xbe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.897762:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x185:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897783:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xbe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075529.897788:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb400/0xe83a7ce271e2ad49 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x184:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea9f0 expref: -99 pid: 67530 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.897795:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92068e00/0xe83a7ce271e257ba lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x59:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe15bc expref: -99 pid: 66136 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.897802:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x184:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897805:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xbc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1769075529.897827:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f6a00/0xe83a7ce271e2accb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x183:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea917 expref: -99 pid: 67524 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897832:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xbc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:3.0:1769075529.897836:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x183:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.897854:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85202000/0xe83a7ce271e256cc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x58:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe1434 expref: -99 pid: 66123 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.897860:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b2200/0xe83a7ce271e2ac8c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x182:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea868 expref: -99 pid: 67519 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1769075529.897864:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xba:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1769075529.897885:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x182:0x0]: losing layout lock 00000080:00200000:0.0:1769075529.897891:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xba:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075529.897908:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b3c00/0xe83a7ce271e2ac15 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x181:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea7a4 expref: -99 pid: 67510 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.897924:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x181:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.897933:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621ae00/0xe83a7ce271e2613e lrc: 1/0,0 mode: --/PW res: [0x280000400:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe262b expref: -99 pid: 66291 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.897940:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83545a00/0xe83a7ce271e2ab6d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x180:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea6e0 expref: -99 pid: 67503 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.897941:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206bc00/0xe83a7ce271e2656d lrc: 1/0,0 mode: --/PW res: [0x280000400:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2c8a expref: -99 pid: 66353 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.897980:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x180:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.897983:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621b600/0xe83a7ce271e26264 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe27c8 expref: -99 pid: 66308 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.897994:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83544400/0xe83a7ce271e2ab2e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x17f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea631 expref: -99 pid: 67497 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898003:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8625d600/0xe83a7ce271e26447 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2aed expref: -99 pid: 66337 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898005:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x17f:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898015:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f7600/0xe83a7ce271e2aaef lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x17e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea582 expref: -99 pid: 67492 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898025:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee2a00/0xe83a7ce271e2638a lrc: 1/0,0 mode: --/PW res: [0x280000400:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2965 expref: -99 pid: 66322 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898028:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x17e:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898043:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f6000/0xe83a7ce271e2aab0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x17d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea4d3 expref: -99 pid: 67487 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898053:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621b200/0xe83a7ce271e25f69 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2330 expref: -99 pid: 66262 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898068:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x17d:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898072:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621bc00/0xe83a7ce271e25eb3 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x61:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe21bd expref: -99 pid: 66250 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898083:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f7c00/0xe83a7ce271e2aa39 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x17c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea40f expref: -99 pid: 67478 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898096:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f07600/0xe83a7ce271e25dfd lrc: 1/0,0 mode: --/PW res: [0x280000400:0x60:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe204a expref: -99 pid: 66235 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898113:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x17c:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898123:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f77000/0xe83a7ce271e25cd7 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x5f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1ead expref: -99 pid: 66221 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898133:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b2a00/0xe83a7ce271e2a991 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x17b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea34b expref: -99 pid: 67470 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898150:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9230a400/0xe83a7ce271e25c59 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x5e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1d4f expref: -99 pid: 66210 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898154:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x17b:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898167:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8600/0xe83a7ce271e2a91a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x17a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea287 expref: -99 pid: 67461 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898171:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81477e00/0xe83a7ce271e25b3a lrc: 1/0,0 mode: --/PW res: [0x280000400:0x5d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1bc7 expref: -99 pid: 66194 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898189:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x17a:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898196:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83547a00/0xe83a7ce271e25a14 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x5c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1a2a expref: -99 pid: 66177 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898202:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745e00/0xe83a7ce271e2a86b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x179:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea1ae expref: -99 pid: 67454 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898212:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83547e00/0xe83a7ce271e2598f lrc: 1/0,0 mode: --/PW res: [0x280000400:0x5b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe18b7 expref: -99 pid: 66167 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898228:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x179:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898234:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83546800/0xe83a7ce271e258d9 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x5a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1744 expref: -99 pid: 66152 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075529.898242:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92068e00/0xe83a7ce271e257ba lrc: 1/0,0 mode: --/PW res: [0x280000400:0x59:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe15bc expref: -99 pid: 66136 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898242:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745000/0xe83a7ce271e2a82c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x178:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea0ff expref: -99 pid: 67449 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898269:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85202000/0xe83a7ce271e256cc lrc: 1/0,0 mode: --/PW res: [0x280000400:0x58:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1434 expref: -99 pid: 66123 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898271:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x178:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898281:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744e00/0xe83a7ce271e2a7b5 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x177:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fea03b expref: -99 pid: 67439 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898289:0:13582:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 16 locks packed 00000080:00200000:3.0:1769075529.898300:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x177:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898314:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744a00/0xe83a7ce271e2a70d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x176:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9f77 expref: -99 pid: 67432 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898317:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621ae00/0xe83a7ce271e2613e lrc: 0/0,0 mode: --/PW res: [0x280000400:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe262b expref: -99 pid: 66291 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898340:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x176:0x0]: losing layout lock 00000100:00080000:1.0:1769075529.898345:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83506a00 x1855007616400640/t0(0) o103->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 440/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1769075529.898362:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206bc00/0xe83a7ce271e2656d lrc: 0/0,0 mode: --/PW res: [0x280000400:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2c8a expref: -99 pid: 66353 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898363:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568000/0xe83a7ce271e2a696 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x175:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9eb3 expref: -99 pid: 67423 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898398:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621b600/0xe83a7ce271e26264 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe27c8 expref: -99 pid: 66308 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898401:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x175:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898424:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8625d600/0xe83a7ce271e26447 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2aed expref: -99 pid: 66337 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898433:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569200/0xe83a7ce271e2a5ee lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x174:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9def expref: -99 pid: 67416 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.898478:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x174:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898482:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee2a00/0xe83a7ce271e2638a lrc: 0/0,0 mode: --/PW res: [0x280000400:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2965 expref: -99 pid: 66322 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898495:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9800/0xe83a7ce271e2a5af lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x173:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9d40 expref: -99 pid: 67410 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898514:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621b200/0xe83a7ce271e25f69 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2330 expref: -99 pid: 66262 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898516:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x173:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898529:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569600/0xe83a7ce271e2a570 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x172:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9c91 expref: -99 pid: 67405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898541:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621bc00/0xe83a7ce271e25eb3 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x61:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe21bd expref: -99 pid: 66250 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898555:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x172:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898559:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f07600/0xe83a7ce271e25dfd lrc: 0/0,0 mode: --/PW res: [0x280000400:0x60:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe204a expref: -99 pid: 66235 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898568:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d25600/0xe83a7ce271e2a531 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x171:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9be2 expref: -99 pid: 67400 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.898591:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x171:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898605:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f77000/0xe83a7ce271e25cd7 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x5f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1ead expref: -99 pid: 66221 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898616:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0c00/0xe83a7ce271e2a4f2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x170:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9b33 expref: -99 pid: 67395 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898620:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9230a400/0xe83a7ce271e25c59 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x5e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1d4f expref: -99 pid: 66210 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898642:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x170:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898649:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81477e00/0xe83a7ce271e25b3a lrc: 0/0,0 mode: --/PW res: [0x280000400:0x5d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1bc7 expref: -99 pid: 66194 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075529.898672:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83547a00/0xe83a7ce271e25a14 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x5c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1a2a expref: -99 pid: 66177 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898672:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569e00/0xe83a7ce271e2a4b3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x16f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9a84 expref: -99 pid: 67389 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.898697:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x16f:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898713:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83547e00/0xe83a7ce271e2598f lrc: 0/0,0 mode: --/PW res: [0x280000400:0x5b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe18b7 expref: -99 pid: 66167 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898718:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568c00/0xe83a7ce271e2a474 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x16e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe99d5 expref: -99 pid: 67384 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898737:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83546800/0xe83a7ce271e258d9 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x5a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1744 expref: -99 pid: 66152 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898749:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x16e:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898755:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92068e00/0xe83a7ce271e257ba lrc: 0/0,0 mode: --/PW res: [0x280000400:0x59:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe15bc expref: -99 pid: 66136 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075529.898767:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9e00/0xe83a7ce271e2a435 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x16d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9926 expref: -99 pid: 67379 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898788:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85202000/0xe83a7ce271e256cc lrc: 0/0,0 mode: --/PW res: [0x280000400:0x58:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe1434 expref: -99 pid: 66123 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1769075529.898792:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x16d:0x0]: losing layout lock 00010000:00010000:0.0:1769075529.898820:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075529.898824:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1400/0xe83a7ce271e2a3be lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x16c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9862 expref: -99 pid: 67370 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1769075529.898825:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075529.898829:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1769075529.898840:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x16c:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898862:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569c00/0xe83a7ce271e2a316 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x16b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe979e expref: -99 pid: 67361 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.898876:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x16b:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898900:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569a00/0xe83a7ce271e2a29f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x16a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe96da expref: -99 pid: 67352 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.898920:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x16a:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898936:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568600/0xe83a7ce271e2a1f7 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x169:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9616 expref: -99 pid: 67345 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.898947:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x169:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898956:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f51400/0xe83a7ce271e2a1b8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x168:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9567 expref: -99 pid: 67340 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.898982:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x168:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.898992:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b6600/0xe83a7ce271e2a141 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x167:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe94a3 expref: -99 pid: 67330 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.898998:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x167:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899004:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569400/0xe83a7ce271e2a099 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x166:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe93df expref: -99 pid: 67323 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899010:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x166:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899017:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568800/0xe83a7ce271e2a022 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x165:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9306 expref: -99 pid: 67314 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899023:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x165:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899030:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9c00/0xe83a7ce271e29f7a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe9257 expref: -99 pid: 67307 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899035:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x164:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899041:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9000/0xe83a7ce271e29f3b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe91a8 expref: -99 pid: 67301 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899046:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x163:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899052:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744800/0xe83a7ce271e29efc lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x162:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe90f9 expref: -99 pid: 67296 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899058:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x162:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899065:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9c00/0xe83a7ce271e29ebd lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x161:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe904a expref: -99 pid: 67291 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899071:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x161:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899081:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9800/0xe83a7ce271e29e7e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x160:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8f9b expref: -99 pid: 67286 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899091:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x160:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899103:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8200/0xe83a7ce271e29e38 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x15f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8ed7 expref: -99 pid: 67280 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899113:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x15f:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899125:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745c00/0xe83a7ce271e29df9 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x15e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8e28 expref: -99 pid: 67275 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899136:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x15e:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899144:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8200/0xe83a7ce271e29d82 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x15d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8d64 expref: -99 pid: 67268 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899150:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x15d:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899158:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8e00/0xe83a7ce271e29d0b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x15c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8ca0 expref: -99 pid: 67259 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899163:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x15c:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899171:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8a00/0xe83a7ce271e29c2b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x15b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8bc7 expref: -99 pid: 67247 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899176:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x15b:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899183:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9200/0xe83a7ce271e29b4b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x15a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8aee expref: -99 pid: 67236 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899188:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x15a:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899196:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f50e00/0xe83a7ce271e29a9c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x159:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8a15 expref: -99 pid: 67229 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899201:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x159:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899208:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561400/0xe83a7ce271e29a5d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x158:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8966 expref: -99 pid: 67224 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899214:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x158:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899220:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2400/0xe83a7ce271e29a1e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x157:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe88b7 expref: -99 pid: 67218 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899226:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x157:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899233:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551a400/0xe83a7ce271e299df lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x156:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8808 expref: -99 pid: 67213 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899238:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x156:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899244:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561e00/0xe83a7ce271e29968 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x155:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x557988ac98fe8744 expref: -99 pid: 67204 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1769075529.899250:0:13603:0:(vvp_object.c:114:vvp_conf_set()) [0x240000403:0x155:0x0]: losing layout lock 00010000:00010000:3.0:1769075529.899268:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83544000/0xe83a7ce271e2b6fe lrc: 1/0,0 mode: --/PR res: [0x240000403:0x19b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98febae4 expref: -99 pid: 67690 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899273:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b922f8c00/0xe83a7ce271e2b6bf lrc: 1/0,0 mode: --/PR res: [0x240000403:0x19a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feba35 expref: -99 pid: 67685 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899278:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f6c00/0xe83a7ce271e2b680 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x199:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb986 expref: -99 pid: 67680 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899282:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83545000/0xe83a7ce271e2b641 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x198:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb8d7 expref: -99 pid: 67675 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899285:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b3e00/0xe83a7ce271e2b5ca lrc: 1/0,0 mode: --/PR res: [0x240000403:0x197:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb813 expref: -99 pid: 67665 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899289:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9234fc00/0xe83a7ce271e2b4ea lrc: 1/0,0 mode: --/PR res: [0x240000403:0x196:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb73a expref: -99 pid: 67654 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899297:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f7400/0xe83a7ce271e2b43b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x195:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb661 expref: -99 pid: 67647 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899302:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834faa00/0xe83a7ce271e2b3fc lrc: 1/0,0 mode: --/PR res: [0x240000403:0x194:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb5b2 expref: -99 pid: 67642 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899306:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d6aa00/0xe83a7ce271e2b3bd lrc: 1/0,0 mode: --/PR res: [0x240000403:0x193:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb503 expref: -99 pid: 67636 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899310:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fa200/0xe83a7ce271e2b377 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x192:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb43f expref: -99 pid: 67631 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899314:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d6b400/0xe83a7ce271e2b300 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x191:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb37b expref: -99 pid: 67622 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899318:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb200/0xe83a7ce271e2b220 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x190:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb2a2 expref: -99 pid: 67611 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899322:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fa000/0xe83a7ce271e2b178 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb1de expref: -99 pid: 67603 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899325:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb800/0xe83a7ce271e2b139 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb12f expref: -99 pid: 67598 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899329:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fae00/0xe83a7ce271e2b0fa lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb080 expref: -99 pid: 67593 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899333:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb000/0xe83a7ce271e2b0bb lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feafd1 expref: -99 pid: 67588 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899341:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fba00/0xe83a7ce271e2b07c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feaf22 expref: -99 pid: 67582 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899346:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83545e00/0xe83a7ce271e2b005 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feae5e expref: -99 pid: 67573 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899350:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fbc00/0xe83a7ce271e2af5d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x189:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fead9a expref: -99 pid: 67566 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899354:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744600/0xe83a7ce271e2af1e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x188:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feaceb expref: -99 pid: 67561 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899358:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b7c00/0xe83a7ce271e2aea7 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x187:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feac27 expref: -99 pid: 67551 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899362:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0da00/0xe83a7ce271e2adff lrc: 1/0,0 mode: --/PR res: [0x240000403:0x186:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feab63 expref: -99 pid: 67544 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899366:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b80030c00/0xe83a7ce271e2adc0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x185:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feaab4 expref: -99 pid: 67539 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899369:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb400/0xe83a7ce271e2ad49 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x184:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea9f0 expref: -99 pid: 67530 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899373:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f6a00/0xe83a7ce271e2accb lrc: 1/0,0 mode: --/PR res: [0x240000403:0x183:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea917 expref: -99 pid: 67524 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899377:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b2200/0xe83a7ce271e2ac8c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x182:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea868 expref: -99 pid: 67519 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899384:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b3c00/0xe83a7ce271e2ac15 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x181:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea7a4 expref: -99 pid: 67510 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899390:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83545a00/0xe83a7ce271e2ab6d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x180:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea6e0 expref: -99 pid: 67503 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899396:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83544400/0xe83a7ce271e2ab2e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x17f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea631 expref: -99 pid: 67497 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899404:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f7600/0xe83a7ce271e2aaef lrc: 1/0,0 mode: --/PR res: [0x240000403:0x17e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea582 expref: -99 pid: 67492 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899410:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f6000/0xe83a7ce271e2aab0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x17d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea4d3 expref: -99 pid: 67487 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899418:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f7c00/0xe83a7ce271e2aa39 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x17c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea40f expref: -99 pid: 67478 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899425:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b2a00/0xe83a7ce271e2a991 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x17b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea34b expref: -99 pid: 67470 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899433:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8600/0xe83a7ce271e2a91a lrc: 1/0,0 mode: --/PR res: [0x240000403:0x17a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea287 expref: -99 pid: 67461 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899440:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745e00/0xe83a7ce271e2a86b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x179:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea1ae expref: -99 pid: 67454 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899446:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745000/0xe83a7ce271e2a82c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x178:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea0ff expref: -99 pid: 67449 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899471:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744e00/0xe83a7ce271e2a7b5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x177:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea03b expref: -99 pid: 67439 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899477:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744a00/0xe83a7ce271e2a70d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x176:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9f77 expref: -99 pid: 67432 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899482:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568000/0xe83a7ce271e2a696 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x175:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9eb3 expref: -99 pid: 67423 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899486:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569200/0xe83a7ce271e2a5ee lrc: 1/0,0 mode: --/PR res: [0x240000403:0x174:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9def expref: -99 pid: 67416 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899490:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9800/0xe83a7ce271e2a5af lrc: 1/0,0 mode: --/PR res: [0x240000403:0x173:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9d40 expref: -99 pid: 67410 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899493:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569600/0xe83a7ce271e2a570 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x172:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9c91 expref: -99 pid: 67405 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899497:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d25600/0xe83a7ce271e2a531 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x171:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9be2 expref: -99 pid: 67400 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899501:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0c00/0xe83a7ce271e2a4f2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x170:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9b33 expref: -99 pid: 67395 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899505:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569e00/0xe83a7ce271e2a4b3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x16f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9a84 expref: -99 pid: 67389 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899509:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568c00/0xe83a7ce271e2a474 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x16e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe99d5 expref: -99 pid: 67384 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899517:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9e00/0xe83a7ce271e2a435 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x16d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9926 expref: -99 pid: 67379 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899521:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1400/0xe83a7ce271e2a3be lrc: 1/0,0 mode: --/PR res: [0x240000403:0x16c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9862 expref: -99 pid: 67370 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899525:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569c00/0xe83a7ce271e2a316 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x16b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe979e expref: -99 pid: 67361 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899529:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569a00/0xe83a7ce271e2a29f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x16a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe96da expref: -99 pid: 67352 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899533:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568600/0xe83a7ce271e2a1f7 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x169:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9616 expref: -99 pid: 67345 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899537:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f51400/0xe83a7ce271e2a1b8 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x168:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9567 expref: -99 pid: 67340 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899541:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b6600/0xe83a7ce271e2a141 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x167:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe94a3 expref: -99 pid: 67330 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899545:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569400/0xe83a7ce271e2a099 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x166:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe93df expref: -99 pid: 67323 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899549:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568800/0xe83a7ce271e2a022 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x165:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9306 expref: -99 pid: 67314 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899553:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9c00/0xe83a7ce271e29f7a lrc: 1/0,0 mode: --/PR res: [0x240000403:0x164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9257 expref: -99 pid: 67307 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899560:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9000/0xe83a7ce271e29f3b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe91a8 expref: -99 pid: 67301 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899564:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744800/0xe83a7ce271e29efc lrc: 1/0,0 mode: --/PR res: [0x240000403:0x162:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe90f9 expref: -99 pid: 67296 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899568:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9c00/0xe83a7ce271e29ebd lrc: 1/0,0 mode: --/PR res: [0x240000403:0x161:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe904a expref: -99 pid: 67291 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899572:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9800/0xe83a7ce271e29e7e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x160:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8f9b expref: -99 pid: 67286 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899576:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8200/0xe83a7ce271e29e38 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x15f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8ed7 expref: -99 pid: 67280 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899580:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745c00/0xe83a7ce271e29df9 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x15e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8e28 expref: -99 pid: 67275 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899584:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8200/0xe83a7ce271e29d82 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x15d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8d64 expref: -99 pid: 67268 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899588:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8e00/0xe83a7ce271e29d0b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x15c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8ca0 expref: -99 pid: 67259 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899592:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8a00/0xe83a7ce271e29c2b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x15b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8bc7 expref: -99 pid: 67247 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899596:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9200/0xe83a7ce271e29b4b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x15a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8aee expref: -99 pid: 67236 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899603:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f50e00/0xe83a7ce271e29a9c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x159:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8a15 expref: -99 pid: 67229 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899607:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561400/0xe83a7ce271e29a5d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x158:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8966 expref: -99 pid: 67224 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899611:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2400/0xe83a7ce271e29a1e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x157:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe88b7 expref: -99 pid: 67218 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899615:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551a400/0xe83a7ce271e299df lrc: 1/0,0 mode: --/PR res: [0x240000403:0x156:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8808 expref: -99 pid: 67213 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899619:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561e00/0xe83a7ce271e29968 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x155:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8744 expref: -99 pid: 67204 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899622:0:13603:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 71 locks packed 00010000:00010000:3.0:1769075529.899636:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83544000/0xe83a7ce271e2b6fe lrc: 0/0,0 mode: --/PR res: [0x240000403:0x19b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98febae4 expref: -99 pid: 67690 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899652:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b922f8c00/0xe83a7ce271e2b6bf lrc: 0/0,0 mode: --/PR res: [0x240000403:0x19a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feba35 expref: -99 pid: 67685 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899663:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f6c00/0xe83a7ce271e2b680 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x199:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb986 expref: -99 pid: 67680 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899678:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83545000/0xe83a7ce271e2b641 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x198:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb8d7 expref: -99 pid: 67675 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899686:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b3e00/0xe83a7ce271e2b5ca lrc: 0/0,0 mode: --/PR res: [0x240000403:0x197:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb813 expref: -99 pid: 67665 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899693:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b9234fc00/0xe83a7ce271e2b4ea lrc: 0/0,0 mode: --/PR res: [0x240000403:0x196:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb73a expref: -99 pid: 67654 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899702:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f7400/0xe83a7ce271e2b43b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x195:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb661 expref: -99 pid: 67647 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899709:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834faa00/0xe83a7ce271e2b3fc lrc: 0/0,0 mode: --/PR res: [0x240000403:0x194:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb5b2 expref: -99 pid: 67642 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899716:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d6aa00/0xe83a7ce271e2b3bd lrc: 0/0,0 mode: --/PR res: [0x240000403:0x193:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb503 expref: -99 pid: 67636 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899722:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fa200/0xe83a7ce271e2b377 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x192:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb43f expref: -99 pid: 67631 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899729:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d6b400/0xe83a7ce271e2b300 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x191:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb37b expref: -99 pid: 67622 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899735:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb200/0xe83a7ce271e2b220 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x190:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb2a2 expref: -99 pid: 67611 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899744:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fa000/0xe83a7ce271e2b178 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb1de expref: -99 pid: 67603 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899755:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb800/0xe83a7ce271e2b139 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb12f expref: -99 pid: 67598 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899766:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fae00/0xe83a7ce271e2b0fa lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feb080 expref: -99 pid: 67593 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899777:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb000/0xe83a7ce271e2b0bb lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feafd1 expref: -99 pid: 67588 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899789:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fba00/0xe83a7ce271e2b07c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feaf22 expref: -99 pid: 67582 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899800:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83545e00/0xe83a7ce271e2b005 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feae5e expref: -99 pid: 67573 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899808:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fbc00/0xe83a7ce271e2af5d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x189:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fead9a expref: -99 pid: 67566 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899815:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744600/0xe83a7ce271e2af1e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x188:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feaceb expref: -99 pid: 67561 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899823:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b7c00/0xe83a7ce271e2aea7 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x187:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feac27 expref: -99 pid: 67551 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899830:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85f0da00/0xe83a7ce271e2adff lrc: 0/0,0 mode: --/PR res: [0x240000403:0x186:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feab63 expref: -99 pid: 67544 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899837:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b80030c00/0xe83a7ce271e2adc0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x185:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98feaab4 expref: -99 pid: 67539 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899844:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834fb400/0xe83a7ce271e2ad49 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x184:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea9f0 expref: -99 pid: 67530 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899850:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f6a00/0xe83a7ce271e2accb lrc: 0/0,0 mode: --/PR res: [0x240000403:0x183:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea917 expref: -99 pid: 67524 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899857:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b2200/0xe83a7ce271e2ac8c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x182:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea868 expref: -99 pid: 67519 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899867:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b3c00/0xe83a7ce271e2ac15 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x181:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea7a4 expref: -99 pid: 67510 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899877:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83545a00/0xe83a7ce271e2ab6d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x180:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea6e0 expref: -99 pid: 67503 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899888:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b83544400/0xe83a7ce271e2ab2e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x17f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea631 expref: -99 pid: 67497 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899899:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f7600/0xe83a7ce271e2aaef lrc: 0/0,0 mode: --/PR res: [0x240000403:0x17e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea582 expref: -99 pid: 67492 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899911:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f6000/0xe83a7ce271e2aab0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x17d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea4d3 expref: -99 pid: 67487 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899921:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834f7c00/0xe83a7ce271e2aa39 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x17c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea40f expref: -99 pid: 67478 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899928:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b2a00/0xe83a7ce271e2a991 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x17b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea34b expref: -99 pid: 67470 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899935:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8600/0xe83a7ce271e2a91a lrc: 0/0,0 mode: --/PR res: [0x240000403:0x17a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea287 expref: -99 pid: 67461 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899941:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745e00/0xe83a7ce271e2a86b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x179:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea1ae expref: -99 pid: 67454 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899949:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745000/0xe83a7ce271e2a82c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x178:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea0ff expref: -99 pid: 67449 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899955:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744e00/0xe83a7ce271e2a7b5 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x177:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fea03b expref: -99 pid: 67439 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899976:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744a00/0xe83a7ce271e2a70d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x176:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9f77 expref: -99 pid: 67432 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899983:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568000/0xe83a7ce271e2a696 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x175:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9eb3 expref: -99 pid: 67423 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899989:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569200/0xe83a7ce271e2a5ee lrc: 0/0,0 mode: --/PR res: [0x240000403:0x174:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9def expref: -99 pid: 67416 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.899996:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9800/0xe83a7ce271e2a5af lrc: 0/0,0 mode: --/PR res: [0x240000403:0x173:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9d40 expref: -99 pid: 67410 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900002:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569600/0xe83a7ce271e2a570 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x172:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9c91 expref: -99 pid: 67405 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900008:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90d25600/0xe83a7ce271e2a531 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x171:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9be2 expref: -99 pid: 67400 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900014:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e0c00/0xe83a7ce271e2a4f2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x170:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9b33 expref: -99 pid: 67395 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900021:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569e00/0xe83a7ce271e2a4b3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x16f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9a84 expref: -99 pid: 67389 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900028:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568c00/0xe83a7ce271e2a474 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x16e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe99d5 expref: -99 pid: 67384 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900034:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9e00/0xe83a7ce271e2a435 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x16d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9926 expref: -99 pid: 67379 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900040:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b834e1400/0xe83a7ce271e2a3be lrc: 0/0,0 mode: --/PR res: [0x240000403:0x16c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9862 expref: -99 pid: 67370 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900046:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569c00/0xe83a7ce271e2a316 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x16b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe979e expref: -99 pid: 67361 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900052:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569a00/0xe83a7ce271e2a29f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x16a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe96da expref: -99 pid: 67352 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900058:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568600/0xe83a7ce271e2a1f7 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x169:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9616 expref: -99 pid: 67345 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900065:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f51400/0xe83a7ce271e2a1b8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x168:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9567 expref: -99 pid: 67340 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900070:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b885b6600/0xe83a7ce271e2a141 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x167:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe94a3 expref: -99 pid: 67330 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900081:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90569400/0xe83a7ce271e2a099 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x166:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe93df expref: -99 pid: 67323 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900092:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90568800/0xe83a7ce271e2a022 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x165:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9306 expref: -99 pid: 67314 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900103:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9c00/0xe83a7ce271e29f7a lrc: 0/0,0 mode: --/PR res: [0x240000403:0x164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe9257 expref: -99 pid: 67307 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900115:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9000/0xe83a7ce271e29f3b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe91a8 expref: -99 pid: 67301 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900126:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92744800/0xe83a7ce271e29efc lrc: 0/0,0 mode: --/PR res: [0x240000403:0x162:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe90f9 expref: -99 pid: 67296 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900137:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9c00/0xe83a7ce271e29ebd lrc: 0/0,0 mode: --/PR res: [0x240000403:0x161:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe904a expref: -99 pid: 67291 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900144:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f9800/0xe83a7ce271e29e7e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x160:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8f9b expref: -99 pid: 67286 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900151:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8200/0xe83a7ce271e29e38 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x15f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8ed7 expref: -99 pid: 67280 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900157:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b92745c00/0xe83a7ce271e29df9 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x15e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8e28 expref: -99 pid: 67275 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900164:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b835f8200/0xe83a7ce271e29d82 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x15d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8d64 expref: -99 pid: 67268 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900171:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8e00/0xe83a7ce271e29d0b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x15c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8ca0 expref: -99 pid: 67259 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900177:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc8a00/0xe83a7ce271e29c2b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x15b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8bc7 expref: -99 pid: 67247 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900184:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b85dc9200/0xe83a7ce271e29b4b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x15a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8aee expref: -99 pid: 67236 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900191:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b87f50e00/0xe83a7ce271e29a9c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x159:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8a15 expref: -99 pid: 67229 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900197:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561400/0xe83a7ce271e29a5d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x158:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8966 expref: -99 pid: 67224 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900203:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b854f2400/0xe83a7ce271e29a1e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x157:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe88b7 expref: -99 pid: 67218 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900209:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b8551a400/0xe83a7ce271e299df lrc: 0/0,0 mode: --/PR res: [0x240000403:0x156:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8808 expref: -99 pid: 67213 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900215:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff995b85db1000 lock: ffff995b90561e00/0xe83a7ce271e29968 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x155:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x557988ac98fe8744 expref: -99 pid: 67204 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1769075529.900222:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075529.900223:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075529.900225:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075530.216569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075530.216608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075530.216623:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075530.920760:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075531.240589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075531.240608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075531.240685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075531.240763:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075531.944899:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075532.264593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075532.264602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075532.264619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075532.264663:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075532.968657:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075533.288596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075533.288597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075533.288597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075533.288636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:3.0:1769075533.992670:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075534.312544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075534.312642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075534.312644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075534.312652:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075534.312728:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616401024:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075534.312953:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616401152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075534.313021:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616401280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075534.313066:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616401408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075534.313103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616401536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075534.313156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616401664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075534.313226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616401792:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075535.016629:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075535.336667:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075535.336768:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075535.336854:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075536.040640:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075536.360528:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075536.360563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075536.360595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075537.064630:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075537.384561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075537.384578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075537.384738:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075537.385015:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075534/real 1769075537] req@ffff995b85acbb80 x1855007616401152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075537.385041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616401152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075537.385056:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075537.385063:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075537.385084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616401152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075537.385095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075534/real 1769075537] req@ffff995b85ac8a80 x1855007616401280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075537.385110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616401280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075537.385121:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075537.385132:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075537.385136:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616401280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075537.385147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075534/real 1769075537] req@ffff995b85acb800 x1855007616401408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075537.385161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616401408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075537.385170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075537.385173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075537.385177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616401408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075537.385184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075534/real 1769075537] req@ffff995b85aca300 x1855007616401024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075537.385198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616401024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075537.385207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075537.385211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075537.385215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616401024:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075537.385247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075534/real 1769075537] req@ffff995b85ac9880 x1855007616401792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075537.385260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616401792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075537.385269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075537.385273:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075537.385277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616401792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075537.385283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075534/real 1769075537] req@ffff995b85acad80 x1855007616401536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075537.385295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acad80 x1855007616401536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075537.385305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075537.385308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075537.385311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616401536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075537.385317:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075534/real 1769075537] req@ffff995b85ac9500 x1855007616401664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075537.385328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9500 x1855007616401664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075537.385336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075537.385339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075537.385342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616401664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075538.408542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075539.112747:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075539.432594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075539.432596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075539.432650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075539.432673:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075539.432682:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075539.432688:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075539.432713:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2525 00000100:00080000:0.0:1769075539.432724:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075539.432731:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075539.432765:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506680 x1855007616401920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075539.432797:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075539.432816:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075539.432818:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075539.432822:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2525 00000100:00080000:0.0:1769075539.432827:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075539.432831:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075539.432839:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075539.432854:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504380 x1855007616402048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075539.432872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075539.432877:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075539.432880:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075539.432885:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2525 00000100:00080000:0.0:1769075539.432889:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075539.432893:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075539.432899:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075539.432912:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616402176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075539.432922:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075539.432925:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075539.432927:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075539.432931:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2525 00000100:00080000:0.0:1769075539.432935:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075539.432942:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075539.432949:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075539.432962:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616402304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075539.432971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075539.432974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075539.432976:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075539.432980:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2525 00000100:00080000:0.0:1769075539.432984:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075539.432988:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075539.432994:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075539.433008:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616402432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075539.433019:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075539.433024:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075539.433028:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075539.433038:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2525 00000100:00080000:0.0:1769075539.433045:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075539.433052:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075539.433061:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075539.433076:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616402560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075539.433086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075539.433090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075539.433093:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075539.433101:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2525 00000100:00080000:0.0:1769075539.433107:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075539.433119:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075539.433125:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075539.433141:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616402688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075540.136654:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 18/18 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1769075540.136715:0:13582:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 18 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1769075540.136727:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747200/0xe83a7ce271e2703a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3e6c expref: -99 pid: 66526 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.136762:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1769075540.136780:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 52/52 locks to regular blp list, 1 blwis in pool 00000080:00200000:0.0:1769075540.136882:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1769075540.136896:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 18/18 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1769075540.136912:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206fe00/0xe83a7ce271e26dc4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3b47 expref: -99 pid: 66489 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.136929:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xee:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075540.136936:0:13603:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 18 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1769075540.136945:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0c600/0xe83a7ce271e271f3 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4335 expref: -99 pid: 66563 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.136957:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xee:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.136966:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075540.136978:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747600/0xe83a7ce271e27319 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4557 expref: -99 pid: 66581 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.136994:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xfc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.137000:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075540.137031:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206ea00/0xe83a7ce271e26ea4 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3c20 expref: -99 pid: 66500 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.137034:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xfc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.137055:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747a00/0xe83a7ce271e270b8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3fca expref: -99 pid: 66537 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137059:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xef:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.137077:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.137094:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xef:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075540.137105:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075540.137108:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee3600/0xe83a7ce271e27400 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4645 expref: -99 pid: 66592 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137124:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xfd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075540.137126:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ee00/0xe83a7ce271e27232 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe43e4 expref: -99 pid: 66568 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137147:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xfd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075540.137162:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xfa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075540.137164:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206e400/0xe83a7ce271e270f7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4079 expref: -99 pid: 66542 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137187:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.137191:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xfa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.137212:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f06600/0xe83a7ce271e26f1b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3ce4 expref: -99 pid: 66510 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137215:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075540.137236:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b84747e00/0xe83a7ce271e27271 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4493 expref: -99 pid: 66573 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.137250:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.137253:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xfb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.137278:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.137284:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xfb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.137296:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83546c00/0xe83a7ce271e271b4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4286 expref: -99 pid: 66558 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.137305:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83546e00/0xe83a7ce271e26fc3 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3da8 expref: -99 pid: 66517 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.137310:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.137321:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.137334:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.137350:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ec00/0xe83a7ce271e27136 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4128 expref: -99 pid: 66547 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137352:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075540.137376:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206f200/0xe83a7ce271e27079 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3f1b expref: -99 pid: 66531 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.137379:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.137407:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.137411:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075540.137422:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d22a00/0xe83a7ce271e26ca5 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe39bf expref: -99 pid: 66473 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.137436:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xec:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.137442:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075540.137504:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206fc00/0xe83a7ce271e27175 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe41d7 expref: -99 pid: 66552 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.137512:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xec:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.137534:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f72a00/0xe83a7ce271e26bbe lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe384c expref: -99 pid: 66460 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137535:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xf7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.137586:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xf7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075540.137589:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xea:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075540.137607:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206f400/0xe83a7ce271e26d1c lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3a83 expref: -99 pid: 66482 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.137629:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xea:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.137630:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xed:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075540.137661:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206e600/0xe83a7ce271e26b40 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe36ee expref: -99 pid: 66450 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137675:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xed:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075540.137693:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075540.137704:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d23600/0xe83a7ce271e26bfd lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe38fb expref: -99 pid: 66465 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137723:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xeb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.137725:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.137748:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206f000/0xe83a7ce271e26a8a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe357b expref: -99 pid: 66437 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137765:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xeb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075540.137819:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075540.137825:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f72000/0xe83a7ce271e26b7f lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe379d expref: -99 pid: 66455 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.137860:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.137871:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.137884:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.137896:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee3c00/0xe83a7ce271e26a05 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3408 expref: -99 pid: 66427 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.137900:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee2c00/0xe83a7ce271e26b01 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe363f expref: -99 pid: 66444 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.137924:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.137931:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.137956:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.137962:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.137984:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d22000/0xe83a7ce271e2694f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3295 expref: -99 pid: 66414 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.137986:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee3a00/0xe83a7ce271e26a4b lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe34cc expref: -99 pid: 66432 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.138012:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.138020:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.138038:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.138047:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.138052:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b87f51c00/0xe83a7ce271e267f8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe30f8 expref: -99 pid: 66393 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138061:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee2800/0xe83a7ce271e269c6 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe3359 expref: -99 pid: 66421 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.138070:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.138078:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.138101:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.138105:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.138121:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee3800/0xe83a7ce271e26670 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe2f5b expref: -99 pid: 66374 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138121:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d22600/0xe83a7ce271e268d8 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe31d1 expref: -99 pid: 66405 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.138149:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xe1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.138170:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xde:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.138174:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xe1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075540.138192:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81477800/0xe83a7ce271e26718 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe301f expref: -99 pid: 66381 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.138197:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xde:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.138202:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xdf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075540.138218:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8625c400/0xe83a7ce271e265eb lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe2de8 expref: -99 pid: 66364 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.138232:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xdf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075540.138237:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xdc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075540.138247:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d22400/0xe83a7ce271e2662a lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe2e97 expref: -99 pid: 66369 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.138261:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xdc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.138266:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xdd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075540.138284:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621b000/0xe83a7ce271e26050 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe24a3 expref: -99 pid: 66275 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075540.138294:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xdd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075540.138314:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8621b800/0xe83a7ce271e260c7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe2567 expref: -99 pid: 66284 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075540.138317:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xd0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075540.138335:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xd1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075540.138349:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xd0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075540.138362:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xd1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075540.138390:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747200/0xe83a7ce271e2703a lrc: 1/0,0 mode: --/PW res: [0x280000400:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3e6c expref: -99 pid: 66526 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138400:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0c600/0xe83a7ce271e271f3 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4335 expref: -99 pid: 66563 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138410:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206fe00/0xe83a7ce271e26dc4 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3b47 expref: -99 pid: 66489 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138435:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747600/0xe83a7ce271e27319 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4557 expref: -99 pid: 66581 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138438:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206ea00/0xe83a7ce271e26ea4 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3c20 expref: -99 pid: 66500 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138447:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747a00/0xe83a7ce271e270b8 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3fca expref: -99 pid: 66537 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138449:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee3600/0xe83a7ce271e27400 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4645 expref: -99 pid: 66592 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138500:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ee00/0xe83a7ce271e27232 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe43e4 expref: -99 pid: 66568 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138510:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206e400/0xe83a7ce271e270f7 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4079 expref: -99 pid: 66542 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138542:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f06600/0xe83a7ce271e26f1b lrc: 1/0,0 mode: --/PW res: [0x280000400:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3ce4 expref: -99 pid: 66510 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138542:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b84747e00/0xe83a7ce271e27271 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4493 expref: -99 pid: 66573 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138604:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83546c00/0xe83a7ce271e271b4 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4286 expref: -99 pid: 66558 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138606:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83546e00/0xe83a7ce271e26fc3 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3da8 expref: -99 pid: 66517 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138630:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ec00/0xe83a7ce271e27136 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4128 expref: -99 pid: 66547 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138633:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206f200/0xe83a7ce271e27079 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3f1b expref: -99 pid: 66531 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138670:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d22a00/0xe83a7ce271e26ca5 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe39bf expref: -99 pid: 66473 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138672:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206fc00/0xe83a7ce271e27175 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe41d7 expref: -99 pid: 66552 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138703:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f72a00/0xe83a7ce271e26bbe lrc: 1/0,0 mode: --/PW res: [0x280000400:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe384c expref: -99 pid: 66460 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138706:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206f400/0xe83a7ce271e26d1c lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3a83 expref: -99 pid: 66482 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138739:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206e600/0xe83a7ce271e26b40 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe36ee expref: -99 pid: 66450 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138743:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d23600/0xe83a7ce271e26bfd lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe38fb expref: -99 pid: 66465 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138773:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206f000/0xe83a7ce271e26a8a lrc: 1/0,0 mode: --/PW res: [0x280000400:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe357b expref: -99 pid: 66437 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138774:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f72000/0xe83a7ce271e26b7f lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe379d expref: -99 pid: 66455 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138833:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee2c00/0xe83a7ce271e26b01 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe363f expref: -99 pid: 66444 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138836:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee3c00/0xe83a7ce271e26a05 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3408 expref: -99 pid: 66427 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138867:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee3a00/0xe83a7ce271e26a4b lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe34cc expref: -99 pid: 66432 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138884:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d22000/0xe83a7ce271e2694f lrc: 1/0,0 mode: --/PW res: [0x280000400:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3295 expref: -99 pid: 66414 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138886:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee2800/0xe83a7ce271e269c6 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3359 expref: -99 pid: 66421 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138910:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b87f51c00/0xe83a7ce271e267f8 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe30f8 expref: -99 pid: 66393 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138911:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d22600/0xe83a7ce271e268d8 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe31d1 expref: -99 pid: 66405 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138933:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81477800/0xe83a7ce271e26718 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe301f expref: -99 pid: 66381 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138942:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee3800/0xe83a7ce271e26670 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2f5b expref: -99 pid: 66374 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138958:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d22400/0xe83a7ce271e2662a lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2e97 expref: -99 pid: 66369 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138969:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8625c400/0xe83a7ce271e265eb lrc: 1/0,0 mode: --/PW res: [0x280000400:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2de8 expref: -99 pid: 66364 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.138982:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8621b800/0xe83a7ce271e260c7 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2567 expref: -99 pid: 66284 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.138997:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621b000/0xe83a7ce271e26050 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe24a3 expref: -99 pid: 66275 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139002:0:13603:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 18 locks packed 00010000:00010000:0.0:1769075540.139009:0:13582:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 18 locks packed 00010000:00010000:0.0:1769075540.139031:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747200/0xe83a7ce271e2703a lrc: 0/0,0 mode: --/PW res: [0x280000400:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3e6c expref: -99 pid: 66526 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139034:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0c600/0xe83a7ce271e271f3 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4335 expref: -99 pid: 66563 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139055:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206fe00/0xe83a7ce271e26dc4 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3b47 expref: -99 pid: 66489 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139079:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206ea00/0xe83a7ce271e26ea4 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3c20 expref: -99 pid: 66500 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139084:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747600/0xe83a7ce271e27319 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4557 expref: -99 pid: 66581 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139128:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee3600/0xe83a7ce271e27400 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4645 expref: -99 pid: 66592 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139133:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747a00/0xe83a7ce271e270b8 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3fca expref: -99 pid: 66537 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139162:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206e400/0xe83a7ce271e270f7 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4079 expref: -99 pid: 66542 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139164:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ee00/0xe83a7ce271e27232 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe43e4 expref: -99 pid: 66568 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139209:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b84747e00/0xe83a7ce271e27271 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4493 expref: -99 pid: 66573 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139210:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f06600/0xe83a7ce271e26f1b lrc: 0/0,0 mode: --/PW res: [0x280000400:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3ce4 expref: -99 pid: 66510 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139243:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83546e00/0xe83a7ce271e26fc3 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3da8 expref: -99 pid: 66517 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139251:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83546c00/0xe83a7ce271e271b4 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4286 expref: -99 pid: 66558 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139274:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206f200/0xe83a7ce271e27079 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3f1b expref: -99 pid: 66531 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139277:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ec00/0xe83a7ce271e27136 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4128 expref: -99 pid: 66547 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139309:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206fc00/0xe83a7ce271e27175 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe41d7 expref: -99 pid: 66552 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139310:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d22a00/0xe83a7ce271e26ca5 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe39bf expref: -99 pid: 66473 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139360:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83f72a00/0xe83a7ce271e26bbe lrc: 0/0,0 mode: --/PW res: [0x280000400:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe384c expref: -99 pid: 66460 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139361:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206f400/0xe83a7ce271e26d1c lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3a83 expref: -99 pid: 66482 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139392:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206e600/0xe83a7ce271e26b40 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe36ee expref: -99 pid: 66450 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139403:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d23600/0xe83a7ce271e26bfd lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe38fb expref: -99 pid: 66465 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139419:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206f000/0xe83a7ce271e26a8a lrc: 0/0,0 mode: --/PW res: [0x280000400:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe357b expref: -99 pid: 66437 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139427:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f72000/0xe83a7ce271e26b7f lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe379d expref: -99 pid: 66455 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139434:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee3c00/0xe83a7ce271e26a05 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3408 expref: -99 pid: 66427 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139466:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d22000/0xe83a7ce271e2694f lrc: 0/0,0 mode: --/PW res: [0x280000400:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3295 expref: -99 pid: 66414 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139495:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee2c00/0xe83a7ce271e26b01 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe363f expref: -99 pid: 66444 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139529:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b87f51c00/0xe83a7ce271e267f8 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe30f8 expref: -99 pid: 66393 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139537:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee3a00/0xe83a7ce271e26a4b lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe34cc expref: -99 pid: 66432 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139594:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee3800/0xe83a7ce271e26670 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2f5b expref: -99 pid: 66374 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139603:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85ee2800/0xe83a7ce271e269c6 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe3359 expref: -99 pid: 66421 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139625:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8625c400/0xe83a7ce271e265eb lrc: 0/0,0 mode: --/PW res: [0x280000400:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2de8 expref: -99 pid: 66364 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139641:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d22600/0xe83a7ce271e268d8 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe31d1 expref: -99 pid: 66405 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139650:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621b000/0xe83a7ce271e26050 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe24a3 expref: -99 pid: 66275 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139701:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81477800/0xe83a7ce271e26718 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe301f expref: -99 pid: 66381 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075540.139704:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075540.139711:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075540.139719:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075540.139739:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90d22400/0xe83a7ce271e2662a lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2e97 expref: -99 pid: 66369 timeout: 0 lvb_type: 1 00000100:00080000:0.0:1769075540.139749:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83505880 x1855007616402816/t0(0) o103->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 456/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1769075540.139756:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8621b800/0xe83a7ce271e260c7 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe2567 expref: -99 pid: 66284 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075540.139832:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075540.139836:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075540.139841:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075540.139847:0:2417:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00010000:00010000:1.0:1769075540.139853:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075540.139859:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b868c6680 x1855007616402944/t0(0) o103->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 456/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075540.140227:0:2419:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b85ac9500 x1855007616403072/t0(0) o103->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 728/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075540.456547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075540.456598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075540.456715:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075541.160581:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075541.480605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075541.480605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075541.480645:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075542.184708:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075542.504592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075542.504592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075542.504660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075542.504698:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075543.208681:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075543.528630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075543.528633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075543.528644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075543.528728:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075544.232777:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075544.552626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075544.552627:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075544.552641:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075544.552648:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075544.552662:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2530 00000100:00080000:0.0:1769075544.552676:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075544.552689:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075544.552701:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075544.552768:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616403200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075544.552806:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075544.552810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075544.552838:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075544.552844:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075544.552860:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2530 00000100:00080000:0.0:1769075544.552866:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075544.552873:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075544.552903:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075544.552931:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616403328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075544.552948:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075544.552952:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075544.552955:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075544.552964:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2530 00000100:00080000:0.0:1769075544.552974:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075544.552983:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075544.552992:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075544.553010:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616403456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075544.553033:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075544.553037:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075544.553039:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075544.553047:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2530 00000100:00080000:0.0:1769075544.553052:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075544.553064:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075544.553078:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075544.553104:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616403584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075544.553189:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075544.553241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075544.553249:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075544.553260:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2530 00000100:00080000:0.0:1769075544.553270:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075544.553285:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075544.553322:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075544.553348:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616403712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075544.553379:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075544.553384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075544.553387:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075544.553421:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2530 00000100:00080000:0.0:1769075544.553434:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075544.553447:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075544.553460:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075544.553480:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616403840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075544.553500:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075544.553562:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075544.553571:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075544.553586:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2530 00000100:00080000:0.0:1769075544.553597:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075544.553612:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075544.553628:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075544.553656:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504380 x1855007616403968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075545.256622:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075545.576558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075545.576724:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075545.576779:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075546.280732:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075546.600992:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075546.601177:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075546.601237:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075546.601255:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075547.304767:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075547.624550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075547.624564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075547.624593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075547.624628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075548.328709:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075548.648556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075548.648556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075548.648583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075548.648622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075549.352821:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 31/31 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1769075549.352916:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 31/62 locks to regular blp list, 2 blwis in pool 00010000:00010000:0.0:1769075549.352975:0:13582:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 31 locks of 31 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1769075549.352995:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8e00/0xe83a7ce271e28858 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6930 expref: -99 pid: 66924 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.353003:0:13603:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 31 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1769075549.353027:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa8600/0xe83a7ce271e28c10 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6fd5 expref: -99 pid: 66985 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.353057:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x12c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.353069:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x135:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.353173:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x12c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.353175:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x135:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.353228:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b854f2200/0xe83a7ce271e28e8d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7394 expref: -99 pid: 67023 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.353242:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f72800/0xe83a7ce271e28b92 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6e77 expref: -99 pid: 66974 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.353248:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x13a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.353281:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x133:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.353286:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x13a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.353308:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8c00/0xe83a7ce271e28b1b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6db3 expref: -99 pid: 66965 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.353324:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x132:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.353330:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x133:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.353354:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x132:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.353369:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa9e00/0xe83a7ce271e28ecc lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7443 expref: -99 pid: 67029 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.353378:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d22800/0xe83a7ce271e28d36 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe71f7 expref: -99 pid: 67003 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.353401:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x13b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.353410:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x138:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.353502:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x13b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.353523:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518200/0xe83a7ce271e28819 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6881 expref: -99 pid: 66918 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.353540:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x12b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.353557:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x138:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.353568:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x12b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.353586:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa5600/0xe83a7ce271e288d6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6a8e expref: -99 pid: 66934 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.353590:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b854f2600/0xe83a7ce271e28e16 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe72d0 expref: -99 pid: 67014 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.353669:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x139:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.353676:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x12e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.353717:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x139:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.353737:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b84746000/0xe83a7ce271e28a73 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6cef expref: -99 pid: 66958 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.353765:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x12e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.353771:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x131:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075549.353796:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ba00/0xe83a7ce271e28c4f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7084 expref: -99 pid: 66990 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.353805:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x131:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.353822:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x136:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.353827:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206a200/0xe83a7ce271e28c8e lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7133 expref: -99 pid: 66995 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.353845:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x137:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.353867:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x136:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.353879:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x137:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.353887:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85202e00/0xe83a7ce271e289fc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6c2b expref: -99 pid: 66951 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.353898:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85b73c00/0xe83a7ce271e28897 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe69df expref: -99 pid: 66929 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.353906:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x130:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.353917:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x12d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.353943:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x130:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.353961:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x12d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.353963:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83546400/0xe83a7ce271e28bd1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6f26 expref: -99 pid: 66980 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.353983:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa9800/0xe83a7ce271e28985 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6b67 expref: -99 pid: 66941 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.353984:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x134:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.354011:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x12f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.354026:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x134:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.354061:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x12f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.354063:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519e00/0xe83a7ce271e287da lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe67d2 expref: -99 pid: 66913 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.354095:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x12a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.354097:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa4800/0xe83a7ce271e28763 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe670e expref: -99 pid: 66904 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.354119:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x129:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.354135:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x12a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.354155:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x129:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.354157:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519a00/0xe83a7ce271e286bb lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe664a expref: -99 pid: 66897 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.354207:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x128:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.354208:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90560c00/0xe83a7ce271e28675 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6586 expref: -99 pid: 66891 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.354230:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x127:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.354242:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x128:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.354262:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa5400/0xe83a7ce271e285fe lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe64c2 expref: -99 pid: 66882 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.354262:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x127:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.354301:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x126:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.354303:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b855c2a00/0xe83a7ce271e2854f lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe63e9 expref: -99 pid: 66875 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.354323:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x125:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.354336:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x126:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.354354:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x125:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.354365:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa4600/0xe83a7ce271e284d8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6325 expref: -99 pid: 66866 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.354379:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518600/0xe83a7ce271e283f8 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe624c expref: -99 pid: 66854 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.354396:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x124:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.354402:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x123:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.354437:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x124:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.354452:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x123:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.354457:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9113aa00/0xe83a7ce271e28350 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe6188 expref: -99 pid: 66847 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.354477:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x122:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.354477:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113b800/0xe83a7ce271e28311 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe60d9 expref: -99 pid: 66842 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.354543:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x122:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.354554:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x121:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075549.354560:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0cc00/0xe83a7ce271e282d2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe602a expref: -99 pid: 66837 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.354579:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x120:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.354605:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x121:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.354614:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x120:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.354655:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113a400/0xe83a7ce271e28293 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5f7b expref: -99 pid: 66831 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.354659:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519200/0xe83a7ce271e28254 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5ecc expref: -99 pid: 66826 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.354696:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x11f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.354700:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x11e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.354745:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x11f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.354749:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x11e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.354771:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8551b000/0xe83a7ce271e281dd lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5e08 expref: -99 pid: 66817 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.354774:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85518000/0xe83a7ce271e28135 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5d44 expref: -99 pid: 66810 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.354806:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x11d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.354811:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x11c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.354853:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x11d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.354862:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x11c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.354872:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518a00/0xe83a7ce271e280f6 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5c95 expref: -99 pid: 66804 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.354883:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0d000/0xe83a7ce271e280b7 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5be6 expref: -99 pid: 66799 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.354890:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x11b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.354925:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x11a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.354940:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x11b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.354959:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x11a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.354960:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0d800/0xe83a7ce271e28078 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5b37 expref: -99 pid: 66794 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.354991:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x119:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075549.354992:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9113ac00/0xe83a7ce271e27e5d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5778 expref: -99 pid: 66758 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.355042:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x114:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.355059:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x119:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.355104:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518c00/0xe83a7ce271e27ffa lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe59d9 expref: -99 pid: 66783 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.355110:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x114:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.355153:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x117:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:0.0:1769075549.355157:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85518e00/0xe83a7ce271e28039 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5a88 expref: -99 pid: 66789 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355180:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x117:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.355183:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x118:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.355199:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0c000/0xe83a7ce271e27c5e lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5517 expref: -99 pid: 66730 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355229:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x111:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.355243:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x118:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.355260:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x111:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.355269:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85518800/0xe83a7ce271e27baf lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe543e expref: -99 pid: 66723 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.355280:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113a600/0xe83a7ce271e27d7d lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe569f expref: -99 pid: 66746 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.355289:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x110:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.355299:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x113:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.355331:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x110:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.355335:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x113:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.355365:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90561000/0xe83a7ce271e27ed4 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe583c expref: -99 pid: 66767 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.355368:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519c00/0xe83a7ce271e27f83 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe5915 expref: -99 pid: 66774 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355389:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x115:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.355397:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x116:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.355419:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x115:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.355433:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x116:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.355448:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0dc00/0xe83a7ce271e27b70 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe538f expref: -99 pid: 66717 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.355460:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84746600/0xe83a7ce271e27cd5 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe55db expref: -99 pid: 66739 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355466:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x10f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.355523:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x112:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.355538:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x10f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.355557:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x112:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.355558:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113b600/0xe83a7ce271e27aba lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe521c expref: -99 pid: 66704 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.355584:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747800/0xe83a7ce271e27af9 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe52cb expref: -99 pid: 66710 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355589:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x10d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.355615:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x10e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.355650:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x10d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.355671:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113b400/0xe83a7ce271e27a04 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe50a9 expref: -99 pid: 66691 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355688:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x10b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.355698:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x10e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.355749:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0de00/0xe83a7ce271e27a7b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe516d expref: -99 pid: 66699 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355753:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x10b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.355787:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x10c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.355792:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323800/0xe83a7ce271e27916 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4f21 expref: -99 pid: 66677 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355813:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x109:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.355838:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x10c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.355861:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x109:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.355862:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81322200/0xe83a7ce271e2798d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4fe5 expref: -99 pid: 66684 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.355897:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x10a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.355901:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90561a00/0xe83a7ce271e277f7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4d99 expref: -99 pid: 66660 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355938:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x107:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.355950:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x10a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.355973:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0ce00/0xe83a7ce271e2789f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4e5d expref: -99 pid: 66668 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.355977:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x107:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.356016:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x108:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.356017:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85202200/0xe83a7ce271e27779 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4c3b expref: -99 pid: 66650 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.356044:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x105:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.356061:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x108:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.356088:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81322400/0xe83a7ce271e277b8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4cea expref: -99 pid: 66655 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.356092:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x105:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.356124:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323a00/0xe83a7ce271e276fb lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4add expref: -99 pid: 66639 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.356126:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x106:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075549.356172:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x103:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.356190:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x106:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.356213:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90560600/0xe83a7ce271e2773a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4b8c expref: -99 pid: 66645 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.356215:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x103:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:0.0:1769075549.356248:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x104:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.356251:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323000/0xe83a7ce271e2767d lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe497f expref: -99 pid: 66629 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.356271:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x101:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.356288:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x104:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.356308:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x101:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.356310:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90561c00/0xe83a7ce271e276bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe4a2e expref: -99 pid: 66634 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.356346:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x102:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.356347:0:13603:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323600/0xe83a7ce271e275c0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe47f7 expref: -99 pid: 66614 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075549.356394:0:13603:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xff:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.356409:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x102:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:1.0:1769075549.356426:0:13603:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xff:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1769075549.356429:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81323e00/0xe83a7ce271e27637 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe48bb expref: -99 pid: 66624 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.356459:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x100:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1769075549.356520:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x100:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.356530:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa8600/0xe83a7ce271e28c10 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6fd5 expref: -99 pid: 66985 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356556:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81322a00/0xe83a7ce271e274e0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe471e expref: -99 pid: 66603 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356563:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f72800/0xe83a7ce271e28b92 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6e77 expref: -99 pid: 66974 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356590:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa9e00/0xe83a7ce271e28ecc lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7443 expref: -99 pid: 67029 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.356592:0:13582:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0xfe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1769075549.356616:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518200/0xe83a7ce271e28819 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6881 expref: -99 pid: 66918 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356651:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b854f2600/0xe83a7ce271e28e16 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe72d0 expref: -99 pid: 67014 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1769075549.356652:0:13582:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0xfe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075549.356693:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b84746000/0xe83a7ce271e28a73 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6cef expref: -99 pid: 66958 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356703:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8e00/0xe83a7ce271e28858 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6930 expref: -99 pid: 66924 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356714:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206a200/0xe83a7ce271e28c8e lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7133 expref: -99 pid: 66995 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356758:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b854f2200/0xe83a7ce271e28e8d lrc: 1/0,0 mode: --/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7394 expref: -99 pid: 67023 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356766:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85b73c00/0xe83a7ce271e28897 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe69df expref: -99 pid: 66929 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356775:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8c00/0xe83a7ce271e28b1b lrc: 1/0,0 mode: --/PW res: [0x280000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6db3 expref: -99 pid: 66965 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356806:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa9800/0xe83a7ce271e28985 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6b67 expref: -99 pid: 66941 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356807:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d22800/0xe83a7ce271e28d36 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe71f7 expref: -99 pid: 67003 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356853:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa5600/0xe83a7ce271e288d6 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6a8e expref: -99 pid: 66934 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356859:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa4800/0xe83a7ce271e28763 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe670e expref: -99 pid: 66904 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356872:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ba00/0xe83a7ce271e28c4f lrc: 1/0,0 mode: --/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7084 expref: -99 pid: 66990 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356877:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90560c00/0xe83a7ce271e28675 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6586 expref: -99 pid: 66891 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356895:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85202e00/0xe83a7ce271e289fc lrc: 1/0,0 mode: --/PW res: [0x280000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6c2b expref: -99 pid: 66951 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356900:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b855c2a00/0xe83a7ce271e2854f lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe63e9 expref: -99 pid: 66875 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356925:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83546400/0xe83a7ce271e28bd1 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6f26 expref: -99 pid: 66980 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356928:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518600/0xe83a7ce271e283f8 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe624c expref: -99 pid: 66854 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.356963:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519e00/0xe83a7ce271e287da lrc: 1/0,0 mode: --/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe67d2 expref: -99 pid: 66913 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.356965:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113b800/0xe83a7ce271e28311 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe60d9 expref: -99 pid: 66842 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357009:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519a00/0xe83a7ce271e286bb lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe664a expref: -99 pid: 66897 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357011:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113a400/0xe83a7ce271e28293 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5f7b expref: -99 pid: 66831 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357062:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa5400/0xe83a7ce271e285fe lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe64c2 expref: -99 pid: 66882 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357063:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8551b000/0xe83a7ce271e281dd lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5e08 expref: -99 pid: 66817 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357101:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518a00/0xe83a7ce271e280f6 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5c95 expref: -99 pid: 66804 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357111:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0d800/0xe83a7ce271e28078 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5b37 expref: -99 pid: 66794 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357120:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa4600/0xe83a7ce271e284d8 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6325 expref: -99 pid: 66866 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357128:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518c00/0xe83a7ce271e27ffa lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe59d9 expref: -99 pid: 66783 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357139:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9113aa00/0xe83a7ce271e28350 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6188 expref: -99 pid: 66847 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357147:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0c000/0xe83a7ce271e27c5e lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5517 expref: -99 pid: 66730 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357159:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0cc00/0xe83a7ce271e282d2 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe602a expref: -99 pid: 66837 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357165:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113a600/0xe83a7ce271e27d7d lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe569f expref: -99 pid: 66746 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357177:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519200/0xe83a7ce271e28254 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5ecc expref: -99 pid: 66826 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357184:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90561000/0xe83a7ce271e27ed4 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe583c expref: -99 pid: 66767 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357194:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85518000/0xe83a7ce271e28135 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5d44 expref: -99 pid: 66810 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357201:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0dc00/0xe83a7ce271e27b70 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe538f expref: -99 pid: 66717 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357211:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0d000/0xe83a7ce271e280b7 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5be6 expref: -99 pid: 66799 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357218:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113b600/0xe83a7ce271e27aba lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe521c expref: -99 pid: 66704 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357229:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9113ac00/0xe83a7ce271e27e5d lrc: 1/0,0 mode: --/PW res: [0x280000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5778 expref: -99 pid: 66758 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357235:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113b400/0xe83a7ce271e27a04 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe50a9 expref: -99 pid: 66691 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357257:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85518e00/0xe83a7ce271e28039 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5a88 expref: -99 pid: 66789 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357261:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323800/0xe83a7ce271e27916 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4f21 expref: -99 pid: 66677 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357294:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85518800/0xe83a7ce271e27baf lrc: 1/0,0 mode: --/PW res: [0x280000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe543e expref: -99 pid: 66723 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357297:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90561a00/0xe83a7ce271e277f7 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4d99 expref: -99 pid: 66660 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357336:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519c00/0xe83a7ce271e27f83 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5915 expref: -99 pid: 66774 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357338:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85202200/0xe83a7ce271e27779 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4c3b expref: -99 pid: 66650 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357393:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323a00/0xe83a7ce271e276fb lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4add expref: -99 pid: 66639 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357397:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84746600/0xe83a7ce271e27cd5 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe55db expref: -99 pid: 66739 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357410:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323000/0xe83a7ce271e2767d lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe497f expref: -99 pid: 66629 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357414:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747800/0xe83a7ce271e27af9 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe52cb expref: -99 pid: 66710 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357434:0:13603:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323600/0xe83a7ce271e275c0 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe47f7 expref: -99 pid: 66614 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357438:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0de00/0xe83a7ce271e27a7b lrc: 1/0,0 mode: --/PW res: [0x280000400:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe516d expref: -99 pid: 66699 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357459:0:13603:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 31 locks packed 00010000:00010000:0.0:1769075549.357463:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81322200/0xe83a7ce271e2798d lrc: 1/0,0 mode: --/PW res: [0x280000400:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4fe5 expref: -99 pid: 66684 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357496:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0ce00/0xe83a7ce271e2789f lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4e5d expref: -99 pid: 66668 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357507:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81322400/0xe83a7ce271e277b8 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4cea expref: -99 pid: 66655 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357516:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90560600/0xe83a7ce271e2773a lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4b8c expref: -99 pid: 66645 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357525:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90561c00/0xe83a7ce271e276bc lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4a2e expref: -99 pid: 66634 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357529:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa8600/0xe83a7ce271e28c10 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6fd5 expref: -99 pid: 66985 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357538:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81323e00/0xe83a7ce271e27637 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe48bb expref: -99 pid: 66624 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357551:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81322a00/0xe83a7ce271e274e0 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe471e expref: -99 pid: 66603 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357559:0:13582:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 31 locks packed 00010000:00010000:1.0:1769075549.357567:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83f72800/0xe83a7ce271e28b92 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6e77 expref: -99 pid: 66974 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357576:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8e00/0xe83a7ce271e28858 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6930 expref: -99 pid: 66924 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357586:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa9e00/0xe83a7ce271e28ecc lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7443 expref: -99 pid: 67029 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357602:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b854f2200/0xe83a7ce271e28e8d lrc: 0/0,0 mode: --/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7394 expref: -99 pid: 67023 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357618:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518200/0xe83a7ce271e28819 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6881 expref: -99 pid: 66918 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357626:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8c00/0xe83a7ce271e28b1b lrc: 0/0,0 mode: --/PW res: [0x280000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6db3 expref: -99 pid: 66965 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357664:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b854f2600/0xe83a7ce271e28e16 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe72d0 expref: -99 pid: 67014 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357671:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d22800/0xe83a7ce271e28d36 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe71f7 expref: -99 pid: 67003 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357692:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b84746000/0xe83a7ce271e28a73 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6cef expref: -99 pid: 66958 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357698:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa5600/0xe83a7ce271e288d6 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6a8e expref: -99 pid: 66934 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357719:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9206a200/0xe83a7ce271e28c8e lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7133 expref: -99 pid: 66995 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357735:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ba00/0xe83a7ce271e28c4f lrc: 0/0,0 mode: --/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7084 expref: -99 pid: 66990 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357771:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85b73c00/0xe83a7ce271e28897 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe69df expref: -99 pid: 66929 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357780:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85202e00/0xe83a7ce271e289fc lrc: 0/0,0 mode: --/PW res: [0x280000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6c2b expref: -99 pid: 66951 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357802:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa9800/0xe83a7ce271e28985 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6b67 expref: -99 pid: 66941 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357813:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83546400/0xe83a7ce271e28bd1 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6f26 expref: -99 pid: 66980 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357839:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b83fa4800/0xe83a7ce271e28763 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe670e expref: -99 pid: 66904 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357868:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519e00/0xe83a7ce271e287da lrc: 0/0,0 mode: --/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe67d2 expref: -99 pid: 66913 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357871:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90560c00/0xe83a7ce271e28675 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6586 expref: -99 pid: 66891 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357911:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b855c2a00/0xe83a7ce271e2854f lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe63e9 expref: -99 pid: 66875 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.357946:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519a00/0xe83a7ce271e286bb lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe664a expref: -99 pid: 66897 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.357954:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518600/0xe83a7ce271e283f8 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe624c expref: -99 pid: 66854 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358024:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113b800/0xe83a7ce271e28311 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe60d9 expref: -99 pid: 66842 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358034:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa5400/0xe83a7ce271e285fe lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe64c2 expref: -99 pid: 66882 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358046:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113a400/0xe83a7ce271e28293 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5f7b expref: -99 pid: 66831 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358063:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa4600/0xe83a7ce271e284d8 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6325 expref: -99 pid: 66866 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358073:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b8551b000/0xe83a7ce271e281dd lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5e08 expref: -99 pid: 66817 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358096:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9113aa00/0xe83a7ce271e28350 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe6188 expref: -99 pid: 66847 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358103:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518a00/0xe83a7ce271e280f6 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5c95 expref: -99 pid: 66804 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358135:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0cc00/0xe83a7ce271e282d2 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe602a expref: -99 pid: 66837 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358140:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0d800/0xe83a7ce271e28078 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5b37 expref: -99 pid: 66794 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358201:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519200/0xe83a7ce271e28254 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5ecc expref: -99 pid: 66826 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358203:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85518c00/0xe83a7ce271e27ffa lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe59d9 expref: -99 pid: 66783 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358267:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85518000/0xe83a7ce271e28135 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5d44 expref: -99 pid: 66810 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358269:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0c000/0xe83a7ce271e27c5e lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5517 expref: -99 pid: 66730 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358349:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0d000/0xe83a7ce271e280b7 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5be6 expref: -99 pid: 66799 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358353:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113a600/0xe83a7ce271e27d7d lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe569f expref: -99 pid: 66746 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358422:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9113ac00/0xe83a7ce271e27e5d lrc: 0/0,0 mode: --/PW res: [0x280000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5778 expref: -99 pid: 66758 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358430:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90561000/0xe83a7ce271e27ed4 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe583c expref: -99 pid: 66767 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358550:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85518e00/0xe83a7ce271e28039 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5a88 expref: -99 pid: 66789 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358559:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85f0dc00/0xe83a7ce271e27b70 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe538f expref: -99 pid: 66717 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358583:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85518800/0xe83a7ce271e27baf lrc: 0/0,0 mode: --/PW res: [0x280000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe543e expref: -99 pid: 66723 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358590:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113b600/0xe83a7ce271e27aba lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe521c expref: -99 pid: 66704 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358671:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b9113b400/0xe83a7ce271e27a04 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe50a9 expref: -99 pid: 66691 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358676:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85519c00/0xe83a7ce271e27f83 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe5915 expref: -99 pid: 66774 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358702:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323800/0xe83a7ce271e27916 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4f21 expref: -99 pid: 66677 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358719:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84746600/0xe83a7ce271e27cd5 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe55db expref: -99 pid: 66739 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358726:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b90561a00/0xe83a7ce271e277f7 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4d99 expref: -99 pid: 66660 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358791:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b85202200/0xe83a7ce271e27779 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4c3b expref: -99 pid: 66650 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358793:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b84747800/0xe83a7ce271e27af9 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe52cb expref: -99 pid: 66710 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358812:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323a00/0xe83a7ce271e276fb lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4add expref: -99 pid: 66639 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358854:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323000/0xe83a7ce271e2767d lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe497f expref: -99 pid: 66629 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358859:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0de00/0xe83a7ce271e27a7b lrc: 0/0,0 mode: --/PW res: [0x280000400:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe516d expref: -99 pid: 66699 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.358892:0:13603:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff995b85db1000 lock: ffff995b81323600/0xe83a7ce271e275c0 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe47f7 expref: -99 pid: 66614 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358911:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81322200/0xe83a7ce271e2798d lrc: 0/0,0 mode: --/PW res: [0x280000400:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4fe5 expref: -99 pid: 66684 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.358944:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85f0ce00/0xe83a7ce271e2789f lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4e5d expref: -99 pid: 66668 timeout: 0 lvb_type: 1 00000100:00080000:1.0:1769075549.358954:0:2418:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b868c7b80 x1855007616404096/t0(0) o103->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 560/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1769075549.358991:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81322400/0xe83a7ce271e277b8 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4cea expref: -99 pid: 66655 timeout: 0 lvb_type: 1 00000100:00100000:1.0:1769075549.359010:0:2418:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:1.0:1769075549.359023:0:2418:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83506680 x1855007616404224/t0(0) o103->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 560/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1769075549.359040:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90560600/0xe83a7ce271e2773a lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4b8c expref: -99 pid: 66645 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.359064:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075549.359069:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075549.359071:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90561c00/0xe83a7ce271e276bc lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe4a2e expref: -99 pid: 66634 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075549.359073:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075549.359100:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81323e00/0xe83a7ce271e27637 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe48bb expref: -99 pid: 66624 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.359122:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b81322a00/0xe83a7ce271e274e0 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe471e expref: -99 pid: 66603 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075549.359155:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075549.359159:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075549.359163:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075549.672559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075549.672607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075549.672656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075549.672660:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075549.672670:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075549.672675:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075549.672687:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2535 00000100:00080000:3.0:1769075549.672714:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075549.672728:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075549.672785:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616404352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075549.672813:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075549.672817:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075549.672819:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075549.672823:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2535 00000100:00080000:3.0:1769075549.672828:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075549.672832:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075549.672841:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075549.672878:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616404480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075549.672896:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075549.672900:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075549.672903:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075549.672907:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2535 00000100:00080000:3.0:1769075549.672912:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075549.672918:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075549.672931:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075549.672944:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616404608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075549.672985:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075549.672990:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075549.672992:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075549.672996:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2535 00000100:00080000:3.0:1769075549.673000:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075549.673005:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075549.673010:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075549.673043:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616404736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075549.673051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075549.673053:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075549.673055:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075549.673062:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2535 00000100:00080000:3.0:1769075549.673070:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075549.673077:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075549.673083:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075549.673119:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616404864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075549.673142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075549.673174:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075549.673179:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075549.673190:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2535 00000100:00080000:3.0:1769075549.673200:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075549.673215:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075549.673237:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075549.673246:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616404992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075549.673253:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075549.673255:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075549.673256:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075549.673261:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2535 00000100:00080000:3.0:1769075549.673268:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075549.673283:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075549.673291:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075549.673312:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616405120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075549.673561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075550.376716:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075550.696568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075550.696841:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075551.400752:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075551.720545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075551.720604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075551.720619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075552.424648:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075552.744660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075552.744660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075552.744661:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075553.448706:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075553.768536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075553.768578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075553.768617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075554.472644:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075554.792540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075554.792598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075554.792621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075554.792628:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075554.792635:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075554.792641:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075554.792653:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2540 00000100:00080000:1.0:1769075554.792664:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075554.792670:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075554.792703:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616405248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075554.792726:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075554.792730:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075554.792732:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075554.792738:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2540 00000100:00080000:1.0:1769075554.792744:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075554.792749:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075554.792759:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075554.792779:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616405376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075554.792796:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075554.792801:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075554.792803:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075554.792809:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2540 00000100:00080000:1.0:1769075554.792815:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075554.792818:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075554.792823:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075554.792833:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616405504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075554.792842:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075554.792844:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075554.792846:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075554.792849:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2540 00000100:00080000:1.0:1769075554.792852:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075554.792854:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075554.792865:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075554.792878:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616405632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075554.792888:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075554.792890:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075554.792891:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075554.792894:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2540 00000100:00080000:1.0:1769075554.792896:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075554.792898:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075554.792903:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075554.792921:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616405760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075554.792965:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075554.792977:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075554.792979:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075554.792985:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2540 00000100:00080000:1.0:1769075554.792989:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075554.792998:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075554.793011:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075554.793024:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616405888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075554.793047:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075554.793058:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075554.793061:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075554.793067:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2540 00000100:00080000:1.0:1769075554.793072:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075554.793077:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075554.793083:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075554.793100:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616406016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075555.496746:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075555.816596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075555.816606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075555.816652:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075556.520744:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075556.840656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075556.840657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075556.840697:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075557.544681:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075557.864624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075557.864631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075557.864687:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075558.568682:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075558.888667:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075558.888668:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075558.888780:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075559.592792:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 74/74 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1769075559.592890:0:68227:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 74 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1769075559.592910:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206e000/0xe83a7ce271e2c170 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fecc41 expref: -99 pid: 67856 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.592963:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1b2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1769075559.592964:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 74/74 locks to regular blp list, 1 blwis in pool 00000080:00200000:1.0:1769075559.593069:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1b2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.593105:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9234ea00/0xe83a7ce271e2b871 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98febd45 expref: -99 pid: 67713 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075559.593142:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1769075559.593145:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x19e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.593210:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x19e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.593244:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b922f9200/0xe83a7ce271e2cce5 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xe2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fee109 expref: -99 pid: 68046 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.593267:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1ce:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.593307:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1ce:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.593352:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545200/0xe83a7ce271e2b6cd lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feba4a expref: -99 pid: 67685 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.593375:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x19a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.593412:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x19a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.593437:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b883af200/0xe83a7ce271e2be36 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fec675 expref: -99 pid: 67802 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.593605:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1aa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.593678:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1aa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.593706:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92068a00/0xe83a7ce271e2b958 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xcb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98febeb8 expref: -99 pid: 67726 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.593741:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1a0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.593786:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1a0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.593809:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545400/0xe83a7ce271e2b752 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98febbbd expref: -99 pid: 67696 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.593831:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x19c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.593874:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x19c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.593896:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7e00/0xe83a7ce271e2aafd lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fea597 expref: -99 pid: 67492 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.593929:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x17e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.593994:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x17e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.594036:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b883ae600/0xe83a7ce271e2bf32 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fec931 expref: -99 pid: 67823 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.594065:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1ae:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.594122:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1ae:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.594155:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b922f9600/0xe83a7ce271e2bb34 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xcd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fec1c8 expref: -99 pid: 67757 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.594196:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1a4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.594247:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1a4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.594270:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ac0400/0xe83a7ce271e2cbe9 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xe0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fede4d expref: -99 pid: 68024 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.594291:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1ca:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.594342:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1ca:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.594364:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b2600/0xe83a7ce271e2b013 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feae73 expref: -99 pid: 67573 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.594386:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x18a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.594445:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x18a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.594522:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85dc9600/0xe83a7ce271e2a928 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fea29c expref: -99 pid: 67461 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.594551:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x17a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.594595:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x17a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.594619:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92745800/0xe83a7ce271e2af2c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xbf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fead00 expref: -99 pid: 67561 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.594654:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x188:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.594704:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x188:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.594727:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fa800/0xe83a7ce271e2c598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fed395 expref: -99 pid: 67923 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.594761:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1bc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.594801:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1bc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.594825:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598000/0xe83a7ce271e2c1ee lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fecd9f expref: -99 pid: 67867 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.594845:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1b4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.594889:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1b4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.594917:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8000/0xe83a7ce271e2c0ba lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fecace expref: -99 pid: 67842 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.594952:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1b0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.595010:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1b0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.595048:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fac00/0xe83a7ce271e2b22e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feb2b7 expref: -99 pid: 67611 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.595096:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x190:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.595151:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x190:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.595182:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83544e00/0xe83a7ce271e2a83a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fea114 expref: -99 pid: 67449 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.595210:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x178:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.595256:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x178:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.595280:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fa400/0xe83a7ce271e2b0c9 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feafe6 expref: -99 pid: 67588 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.595308:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x18c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.595361:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x18c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.595383:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b922f9000/0xe83a7ce271e2ba15 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xcc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fec040 expref: -99 pid: 67740 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.595402:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1a2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.595524:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1a2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.595562:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b833b0800/0xe83a7ce271e2cc67 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xe1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fedfab expref: -99 pid: 68036 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.595611:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1cc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.595650:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1cc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.595669:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b3800/0xe83a7ce271e2ab7b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xbb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fea6f5 expref: -99 pid: 67503 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.595714:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x180:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.595768:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x180:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.595790:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ac1200/0xe83a7ce271e2cb2c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xdf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fedcc5 expref: -99 pid: 68012 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.595821:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1c8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.595870:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1c8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.595891:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83512400/0xe83a7ce271e2ca3e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xde:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fedb3d expref: -99 pid: 67995 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.595927:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1c6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.595992:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1c6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.596038:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598800/0xe83a7ce271e2c87e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xdd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fed98b expref: -99 pid: 67975 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.596074:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1c4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.596125:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1c4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.596159:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598a00/0xe83a7ce271e2c7f9 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xdc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fed818 expref: -99 pid: 67964 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.596194:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1c2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.596244:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1c2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.596267:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90568a00/0xe83a7ce271e2c774 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xdb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fed6a5 expref: -99 pid: 67954 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.596295:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1c0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.596336:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1c0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.596358:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545c00/0xe83a7ce271e2c64e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xda:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fed508 expref: -99 pid: 67937 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.596378:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1be:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.596428:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1be:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.596449:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598400/0xe83a7ce271e2c4b1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fed222 expref: -99 pid: 67910 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.596535:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1ba:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.596581:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1ba:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.596606:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7800/0xe83a7ce271e2b64f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feb8ec expref: -99 pid: 67675 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.596646:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x198:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.596693:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x198:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.596726:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ac0000/0xe83a7ce271e2c38b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fed085 expref: -99 pid: 67894 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.596743:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1b8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.596781:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1b8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.596807:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b3600/0xe83a7ce271e2ae0d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xbe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feab78 expref: -99 pid: 67544 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.596847:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x186:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.596886:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x186:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.596912:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598e00/0xe83a7ce271e2c2d5 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fecf12 expref: -99 pid: 67879 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.596951:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1b6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.597011:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1b6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.597050:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545800/0xe83a7ce271e2a71b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe9f8c expref: -99 pid: 67432 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.597110:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x176:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.597166:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x176:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.597199:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f6200/0xe83a7ce271e2b40a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feb5c7 expref: -99 pid: 67642 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.597223:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x194:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.597265:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x194:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.597289:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fb600/0xe83a7ce271e2ad57 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xbd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feaa05 expref: -99 pid: 67530 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.597334:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x184:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.597376:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x184:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.597406:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9234f600/0xe83a7ce271e2b385 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feb454 expref: -99 pid: 67631 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.597436:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x192:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.597546:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x192:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.597568:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7a00/0xe83a7ce271e2aa47 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fea424 expref: -99 pid: 67478 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.597592:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x17c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.597650:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x17c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.597672:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d6bc00/0xe83a7ce271e2beb4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xd1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fec7d3 expref: -99 pid: 67813 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.597700:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1ac:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.597736:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1ac:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.597760:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7000/0xe83a7ce271e2ac9a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xbc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fea87d expref: -99 pid: 67519 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.597778:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x182:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.597818:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x182:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.597843:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b883afa00/0xe83a7ce271e2bcdf lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xcf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fec4d8 expref: -99 pid: 67784 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.597860:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1a8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.597900:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1a8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.597932:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b3200/0xe83a7ce271e2b147 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feb144 expref: -99 pid: 67598 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.597980:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x18e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.598040:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x18e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.598074:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d6ae00/0xe83a7ce271e2bc1b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fec33b expref: -99 pid: 67769 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.598132:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x1a6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.598190:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x1a6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.598229:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9234e400/0xe83a7ce271e2b4f8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xc6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98feb74f expref: -99 pid: 67654 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.598247:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x196:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.598288:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x196:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.598315:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8200/0xe83a7ce271e2a5fc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe9e04 expref: -99 pid: 67416 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.598332:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x174:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.598371:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x174:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.598396:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90568e00/0xe83a7ce271e2a57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe9ca6 expref: -99 pid: 67405 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.598413:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x172:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.598452:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x172:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.598531:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90568400/0xe83a7ce271e2a500 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe9b48 expref: -99 pid: 67395 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.598558:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x170:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.598603:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x170:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.598624:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834e0a00/0xe83a7ce271e2a482 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe99ea expref: -99 pid: 67384 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.598645:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x16e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.598694:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x16e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.598716:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834e0800/0xe83a7ce271e2a3cc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe9877 expref: -99 pid: 67370 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.598750:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x16c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.598787:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x16c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.598808:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b2e00/0xe83a7ce271e2a2ad lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xb0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe96ef expref: -99 pid: 67352 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.598843:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x16a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.598884:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x16a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.598908:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b2800/0xe83a7ce271e2a1c6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xaf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe957c expref: -99 pid: 67340 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.598948:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x168:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.599014:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x168:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.599066:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206b000/0xe83a7ce271e2a0a7 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xae:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe93f4 expref: -99 pid: 67323 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.599146:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x166:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.599199:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x166:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.599233:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85dc8000/0xe83a7ce271e29f88 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xad:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe926c expref: -99 pid: 67307 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.599253:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x164:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.599302:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x164:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.599328:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa9600/0xe83a7ce271e29f0a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xac:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe910e expref: -99 pid: 67296 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.599346:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x162:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.599396:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x162:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.599422:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85dc9400/0xe83a7ce271e29d19 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe8cb5 expref: -99 pid: 67259 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.599439:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x15c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.599534:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x15c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.599557:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b6000/0xe83a7ce271e297e0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe850d expref: -99 pid: 67182 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.599587:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x152:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.599637:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x152:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.599660:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92744400/0xe83a7ce271e29e8c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe8fb0 expref: -99 pid: 67286 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.599688:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x160:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.599738:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x160:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.599759:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621a800/0xe83a7ce271e299ed lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe881d expref: -99 pid: 67213 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.599781:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x156:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.599830:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x156:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.599851:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f9a00/0xe83a7ce271e29e07 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xaa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe8e3d expref: -99 pid: 67275 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.599886:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x15e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.599930:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x15e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.599960:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa4200/0xe83a7ce271e296dd lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe823c expref: -99 pid: 67161 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600002:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x14e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.600060:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x14e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.600100:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f8a00/0xe83a7ce271e29896 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe8680 expref: -99 pid: 67195 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600129:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x154:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.600181:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x154:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.600211:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b854f3600/0xe83a7ce271e29b59 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe8b03 expref: -99 pid: 67236 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600248:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x15a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.600303:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x15a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.600324:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b7200/0xe83a7ce271e2975b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe839a expref: -99 pid: 67172 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600351:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x150:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.600389:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x150:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.600410:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee3200/0xe83a7ce271e29a6b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe897b expref: -99 pid: 67224 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600445:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x158:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.600539:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x158:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.600566:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa5e00/0xe83a7ce271e2965f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe80de expref: -99 pid: 67151 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600589:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x14c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.600637:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x14c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.600662:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92745a00/0xe83a7ce271e29539 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7f41 expref: -99 pid: 67132 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600699:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x14a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.600737:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x14a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.600762:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ae00/0xe83a7ce271e29379 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7d8f expref: -99 pid: 67110 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600785:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x148:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.600823:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x148:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.600844:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa9a00/0xe83a7ce271e29292 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7c1c expref: -99 pid: 67097 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600872:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x146:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.600902:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x146:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.600931:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa9200/0xe83a7ce271e29214 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7abe expref: -99 pid: 67087 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.600985:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x144:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.601030:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x144:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.601080:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834e0600/0xe83a7ce271e29196 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7960 expref: -99 pid: 67076 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.601134:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x142:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.601180:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x142:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.601210:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f9000/0xe83a7ce271e29118 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7802 expref: -99 pid: 67066 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.601262:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x140:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.601303:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x140:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.601326:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b854f2a00/0xe83a7ce271e28fc1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe7665 expref: -99 pid: 67045 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.601391:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x13e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.601427:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x13e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.601449:0:68227:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f8000/0xe83a7ce271e28f0b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x557988ac98fe74f2 expref: -99 pid: 67035 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1769075559.601524:0:68227:0:(vvp_io.c:1854:vvp_io_init()) [0x240000403:0x13c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1769075559.601555:0:68227:0:(vvp_io.c:289:vvp_io_fini()) [0x240000403:0x13c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1769075559.601608:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206e000/0xe83a7ce271e2c170 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fecc41 expref: -99 pid: 67856 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601658:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9234ea00/0xe83a7ce271e2b871 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98febd45 expref: -99 pid: 67713 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601699:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b922f9200/0xe83a7ce271e2cce5 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xe2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fee109 expref: -99 pid: 68046 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601716:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545200/0xe83a7ce271e2b6cd lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feba4a expref: -99 pid: 67685 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601737:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b883af200/0xe83a7ce271e2be36 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec675 expref: -99 pid: 67802 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601768:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92068a00/0xe83a7ce271e2b958 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xcb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98febeb8 expref: -99 pid: 67726 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601804:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545400/0xe83a7ce271e2b752 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98febbbd expref: -99 pid: 67696 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601845:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7e00/0xe83a7ce271e2aafd lrc: 1/0,0 mode: --/PW res: [0x280000400:0xba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea597 expref: -99 pid: 67492 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601886:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b883ae600/0xe83a7ce271e2bf32 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec931 expref: -99 pid: 67823 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601932:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b922f9600/0xe83a7ce271e2bb34 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xcd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec1c8 expref: -99 pid: 67757 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.601965:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ac0400/0xe83a7ce271e2cbe9 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xe0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fede4d expref: -99 pid: 68024 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602014:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b2600/0xe83a7ce271e2b013 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feae73 expref: -99 pid: 67573 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602053:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85dc9600/0xe83a7ce271e2a928 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea29c expref: -99 pid: 67461 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602093:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92745800/0xe83a7ce271e2af2c lrc: 1/0,0 mode: --/PW res: [0x280000400:0xbf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fead00 expref: -99 pid: 67561 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602128:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fa800/0xe83a7ce271e2c598 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed395 expref: -99 pid: 67923 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602161:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598000/0xe83a7ce271e2c1ee lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fecd9f expref: -99 pid: 67867 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602174:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8000/0xe83a7ce271e2c0ba lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fecace expref: -99 pid: 67842 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602220:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fac00/0xe83a7ce271e2b22e lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb2b7 expref: -99 pid: 67611 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602258:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83544e00/0xe83a7ce271e2a83a lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea114 expref: -99 pid: 67449 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602273:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fa400/0xe83a7ce271e2b0c9 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feafe6 expref: -99 pid: 67588 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602298:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b922f9000/0xe83a7ce271e2ba15 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xcc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec040 expref: -99 pid: 67740 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602338:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b833b0800/0xe83a7ce271e2cc67 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xe1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fedfab expref: -99 pid: 68036 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602378:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b3800/0xe83a7ce271e2ab7b lrc: 1/0,0 mode: --/PW res: [0x280000400:0xbb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea6f5 expref: -99 pid: 67503 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602417:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ac1200/0xe83a7ce271e2cb2c lrc: 1/0,0 mode: --/PW res: [0x280000400:0xdf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fedcc5 expref: -99 pid: 68012 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602453:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83512400/0xe83a7ce271e2ca3e lrc: 1/0,0 mode: --/PW res: [0x280000400:0xde:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fedb3d expref: -99 pid: 67995 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602525:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598800/0xe83a7ce271e2c87e lrc: 1/0,0 mode: --/PW res: [0x280000400:0xdd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed98b expref: -99 pid: 67975 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602552:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598a00/0xe83a7ce271e2c7f9 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xdc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed818 expref: -99 pid: 67964 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602582:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90568a00/0xe83a7ce271e2c774 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xdb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed6a5 expref: -99 pid: 67954 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602613:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545c00/0xe83a7ce271e2c64e lrc: 1/0,0 mode: --/PW res: [0x280000400:0xda:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed508 expref: -99 pid: 67937 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602644:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598400/0xe83a7ce271e2c4b1 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed222 expref: -99 pid: 67910 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602674:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7800/0xe83a7ce271e2b64f lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb8ec expref: -99 pid: 67675 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602699:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ac0000/0xe83a7ce271e2c38b lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed085 expref: -99 pid: 67894 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602722:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b3600/0xe83a7ce271e2ae0d lrc: 1/0,0 mode: --/PW res: [0x280000400:0xbe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feab78 expref: -99 pid: 67544 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602740:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598e00/0xe83a7ce271e2c2d5 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fecf12 expref: -99 pid: 67879 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602757:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545800/0xe83a7ce271e2a71b lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9f8c expref: -99 pid: 67432 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602775:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f6200/0xe83a7ce271e2b40a lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb5c7 expref: -99 pid: 67642 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602801:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fb600/0xe83a7ce271e2ad57 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xbd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feaa05 expref: -99 pid: 67530 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602816:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9234f600/0xe83a7ce271e2b385 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb454 expref: -99 pid: 67631 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602825:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7a00/0xe83a7ce271e2aa47 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea424 expref: -99 pid: 67478 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602834:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d6bc00/0xe83a7ce271e2beb4 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xd1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec7d3 expref: -99 pid: 67813 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602849:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7000/0xe83a7ce271e2ac9a lrc: 1/0,0 mode: --/PW res: [0x280000400:0xbc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea87d expref: -99 pid: 67519 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602867:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b883afa00/0xe83a7ce271e2bcdf lrc: 1/0,0 mode: --/PW res: [0x280000400:0xcf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec4d8 expref: -99 pid: 67784 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602878:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b3200/0xe83a7ce271e2b147 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb144 expref: -99 pid: 67598 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.602900:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d6ae00/0xe83a7ce271e2bc1b lrc: 1/0,0 mode: --/PW res: [0x280000400:0xce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec33b expref: -99 pid: 67769 timeout: 0 lvb_type: 1 00000100:00080000:3.0:1769075559.602912:0:2420:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b85acad80 x1855007616406144/t0(0) o103->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 904/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1769075559.602961:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9234e400/0xe83a7ce271e2b4f8 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xc6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb74f expref: -99 pid: 67654 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603045:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8200/0xe83a7ce271e2a5fc lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9e04 expref: -99 pid: 67416 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603067:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90568e00/0xe83a7ce271e2a57e lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9ca6 expref: -99 pid: 67405 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603101:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90568400/0xe83a7ce271e2a500 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9b48 expref: -99 pid: 67395 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603115:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834e0a00/0xe83a7ce271e2a482 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe99ea expref: -99 pid: 67384 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603139:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834e0800/0xe83a7ce271e2a3cc lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9877 expref: -99 pid: 67370 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603152:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b2e00/0xe83a7ce271e2a2ad lrc: 1/0,0 mode: --/PW res: [0x280000400:0xb0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe96ef expref: -99 pid: 67352 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603166:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b2800/0xe83a7ce271e2a1c6 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xaf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe957c expref: -99 pid: 67340 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603201:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206b000/0xe83a7ce271e2a0a7 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xae:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe93f4 expref: -99 pid: 67323 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603212:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85dc8000/0xe83a7ce271e29f88 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xad:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe926c expref: -99 pid: 67307 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603224:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa9600/0xe83a7ce271e29f0a lrc: 1/0,0 mode: --/PW res: [0x280000400:0xac:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe910e expref: -99 pid: 67296 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603260:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85dc9400/0xe83a7ce271e29d19 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8cb5 expref: -99 pid: 67259 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603275:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b6000/0xe83a7ce271e297e0 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe850d expref: -99 pid: 67182 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603320:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92744400/0xe83a7ce271e29e8c lrc: 1/0,0 mode: --/PW res: [0x280000400:0xab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8fb0 expref: -99 pid: 67286 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603333:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621a800/0xe83a7ce271e299ed lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe881d expref: -99 pid: 67213 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603374:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f9a00/0xe83a7ce271e29e07 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xaa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8e3d expref: -99 pid: 67275 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603387:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa4200/0xe83a7ce271e296dd lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe823c expref: -99 pid: 67161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603418:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f8a00/0xe83a7ce271e29896 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8680 expref: -99 pid: 67195 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603438:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b854f3600/0xe83a7ce271e29b59 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8b03 expref: -99 pid: 67236 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603452:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b7200/0xe83a7ce271e2975b lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe839a expref: -99 pid: 67172 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603510:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee3200/0xe83a7ce271e29a6b lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe897b expref: -99 pid: 67224 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603520:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa5e00/0xe83a7ce271e2965f lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe80de expref: -99 pid: 67151 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603530:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92745a00/0xe83a7ce271e29539 lrc: 1/0,0 mode: --/PW res: [0x280000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7f41 expref: -99 pid: 67132 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603544:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ae00/0xe83a7ce271e29379 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7d8f expref: -99 pid: 67110 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603582:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa9a00/0xe83a7ce271e29292 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7c1c expref: -99 pid: 67097 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603597:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa9200/0xe83a7ce271e29214 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7abe expref: -99 pid: 67087 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603610:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834e0600/0xe83a7ce271e29196 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7960 expref: -99 pid: 67076 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603620:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f9000/0xe83a7ce271e29118 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7802 expref: -99 pid: 67066 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603642:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b854f2a00/0xe83a7ce271e28fc1 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7665 expref: -99 pid: 67045 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603654:0:68227:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f8000/0xe83a7ce271e28f0b lrc: 1/0,0 mode: --/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe74f2 expref: -99 pid: 67035 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603677:0:68227:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 74 locks packed 00010000:00010000:1.0:1769075559.603725:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206e000/0xe83a7ce271e2c170 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fecc41 expref: -99 pid: 67856 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603761:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9234ea00/0xe83a7ce271e2b871 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98febd45 expref: -99 pid: 67713 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.603846:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b922f9200/0xe83a7ce271e2cce5 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xe2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fee109 expref: -99 pid: 68046 timeout: 0 lvb_type: 1 00000100:00080000:0.0:1769075559.603861:0:2418:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b868c4e00 x1855007616406272/t0(0) o103->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:17/18 lens 904/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1769075559.603939:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545200/0xe83a7ce271e2b6cd lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feba4a expref: -99 pid: 67685 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604010:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b883af200/0xe83a7ce271e2be36 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec675 expref: -99 pid: 67802 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604052:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92068a00/0xe83a7ce271e2b958 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xcb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98febeb8 expref: -99 pid: 67726 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604121:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545400/0xe83a7ce271e2b752 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98febbbd expref: -99 pid: 67696 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604176:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7e00/0xe83a7ce271e2aafd lrc: 0/0,0 mode: --/PW res: [0x280000400:0xba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea597 expref: -99 pid: 67492 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604205:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b883ae600/0xe83a7ce271e2bf32 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec931 expref: -99 pid: 67823 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604269:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b922f9600/0xe83a7ce271e2bb34 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xcd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec1c8 expref: -99 pid: 67757 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604340:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ac0400/0xe83a7ce271e2cbe9 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xe0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fede4d expref: -99 pid: 68024 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604415:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b2600/0xe83a7ce271e2b013 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feae73 expref: -99 pid: 67573 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604544:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85dc9600/0xe83a7ce271e2a928 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea29c expref: -99 pid: 67461 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604574:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92745800/0xe83a7ce271e2af2c lrc: 0/0,0 mode: --/PW res: [0x280000400:0xbf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fead00 expref: -99 pid: 67561 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604626:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fa800/0xe83a7ce271e2c598 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed395 expref: -99 pid: 67923 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604650:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598000/0xe83a7ce271e2c1ee lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fecd9f expref: -99 pid: 67867 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604676:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8000/0xe83a7ce271e2c0ba lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fecace expref: -99 pid: 67842 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604703:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fac00/0xe83a7ce271e2b22e lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb2b7 expref: -99 pid: 67611 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604730:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83544e00/0xe83a7ce271e2a83a lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea114 expref: -99 pid: 67449 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604756:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fa400/0xe83a7ce271e2b0c9 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feafe6 expref: -99 pid: 67588 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604815:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b922f9000/0xe83a7ce271e2ba15 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xcc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec040 expref: -99 pid: 67740 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604883:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b833b0800/0xe83a7ce271e2cc67 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xe1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fedfab expref: -99 pid: 68036 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.604963:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b3800/0xe83a7ce271e2ab7b lrc: 0/0,0 mode: --/PW res: [0x280000400:0xbb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea6f5 expref: -99 pid: 67503 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605036:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ac1200/0xe83a7ce271e2cb2c lrc: 0/0,0 mode: --/PW res: [0x280000400:0xdf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fedcc5 expref: -99 pid: 68012 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605063:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83512400/0xe83a7ce271e2ca3e lrc: 0/0,0 mode: --/PW res: [0x280000400:0xde:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fedb3d expref: -99 pid: 67995 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605090:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598800/0xe83a7ce271e2c87e lrc: 0/0,0 mode: --/PW res: [0x280000400:0xdd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed98b expref: -99 pid: 67975 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605116:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598a00/0xe83a7ce271e2c7f9 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xdc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed818 expref: -99 pid: 67964 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605142:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90568a00/0xe83a7ce271e2c774 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xdb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed6a5 expref: -99 pid: 67954 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605169:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545c00/0xe83a7ce271e2c64e lrc: 0/0,0 mode: --/PW res: [0x280000400:0xda:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed508 expref: -99 pid: 67937 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605234:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598400/0xe83a7ce271e2c4b1 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed222 expref: -99 pid: 67910 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605276:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7800/0xe83a7ce271e2b64f lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb8ec expref: -99 pid: 67675 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605302:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ac0000/0xe83a7ce271e2c38b lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fed085 expref: -99 pid: 67894 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605336:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b3600/0xe83a7ce271e2ae0d lrc: 0/0,0 mode: --/PW res: [0x280000400:0xbe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feab78 expref: -99 pid: 67544 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605383:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b86598e00/0xe83a7ce271e2c2d5 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fecf12 expref: -99 pid: 67879 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605408:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83545800/0xe83a7ce271e2a71b lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9f8c expref: -99 pid: 67432 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605442:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f6200/0xe83a7ce271e2b40a lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb5c7 expref: -99 pid: 67642 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605526:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834fb600/0xe83a7ce271e2ad57 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xbd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feaa05 expref: -99 pid: 67530 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605554:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9234f600/0xe83a7ce271e2b385 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb454 expref: -99 pid: 67631 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605591:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7a00/0xe83a7ce271e2aa47 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea424 expref: -99 pid: 67478 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605618:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d6bc00/0xe83a7ce271e2beb4 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xd1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec7d3 expref: -99 pid: 67813 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605644:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834f7000/0xe83a7ce271e2ac9a lrc: 0/0,0 mode: --/PW res: [0x280000400:0xbc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fea87d expref: -99 pid: 67519 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605671:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b883afa00/0xe83a7ce271e2bcdf lrc: 0/0,0 mode: --/PW res: [0x280000400:0xcf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec4d8 expref: -99 pid: 67784 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605705:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b3200/0xe83a7ce271e2b147 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb144 expref: -99 pid: 67598 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605753:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90d6ae00/0xe83a7ce271e2bc1b lrc: 0/0,0 mode: --/PW res: [0x280000400:0xce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fec33b expref: -99 pid: 67769 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605777:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9234e400/0xe83a7ce271e2b4f8 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xc6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98feb74f expref: -99 pid: 67654 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605804:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa8200/0xe83a7ce271e2a5fc lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9e04 expref: -99 pid: 67416 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605830:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90568e00/0xe83a7ce271e2a57e lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9ca6 expref: -99 pid: 67405 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605863:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b90568400/0xe83a7ce271e2a500 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9b48 expref: -99 pid: 67395 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605889:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834e0a00/0xe83a7ce271e2a482 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe99ea expref: -99 pid: 67384 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605922:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834e0800/0xe83a7ce271e2a3cc lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe9877 expref: -99 pid: 67370 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.605999:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b2e00/0xe83a7ce271e2a2ad lrc: 0/0,0 mode: --/PW res: [0x280000400:0xb0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe96ef expref: -99 pid: 67352 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606041:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b2800/0xe83a7ce271e2a1c6 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xaf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe957c expref: -99 pid: 67340 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606069:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206b000/0xe83a7ce271e2a0a7 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xae:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe93f4 expref: -99 pid: 67323 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606095:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85dc8000/0xe83a7ce271e29f88 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xad:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe926c expref: -99 pid: 67307 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606122:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa9600/0xe83a7ce271e29f0a lrc: 0/0,0 mode: --/PW res: [0x280000400:0xac:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe910e expref: -99 pid: 67296 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606149:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85dc9400/0xe83a7ce271e29d19 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8cb5 expref: -99 pid: 67259 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606181:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b6000/0xe83a7ce271e297e0 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe850d expref: -99 pid: 67182 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606227:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92744400/0xe83a7ce271e29e8c lrc: 0/0,0 mode: --/PW res: [0x280000400:0xab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8fb0 expref: -99 pid: 67286 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606255:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b8621a800/0xe83a7ce271e299ed lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe881d expref: -99 pid: 67213 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606293:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f9a00/0xe83a7ce271e29e07 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xaa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8e3d expref: -99 pid: 67275 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606356:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa4200/0xe83a7ce271e296dd lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe823c expref: -99 pid: 67161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606383:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f8a00/0xe83a7ce271e29896 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8680 expref: -99 pid: 67195 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606406:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b854f3600/0xe83a7ce271e29b59 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe8b03 expref: -99 pid: 67236 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606420:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b885b7200/0xe83a7ce271e2975b lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe839a expref: -99 pid: 67172 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606433:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b85ee3200/0xe83a7ce271e29a6b lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe897b expref: -99 pid: 67224 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606452:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa5e00/0xe83a7ce271e2965f lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe80de expref: -99 pid: 67151 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606519:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b92745a00/0xe83a7ce271e29539 lrc: 0/0,0 mode: --/PW res: [0x280000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7f41 expref: -99 pid: 67132 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606536:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b9206ae00/0xe83a7ce271e29379 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7d8f expref: -99 pid: 67110 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606550:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa9a00/0xe83a7ce271e29292 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7c1c expref: -99 pid: 67097 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606563:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b83fa9200/0xe83a7ce271e29214 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7abe expref: -99 pid: 67087 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606575:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b834e0600/0xe83a7ce271e29196 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7960 expref: -99 pid: 67076 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606587:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f9000/0xe83a7ce271e29118 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7802 expref: -99 pid: 67066 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606599:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b854f2a00/0xe83a7ce271e28fc1 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe7665 expref: -99 pid: 67045 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606611:0:68227:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff995b85db1000 lock: ffff995b835f8000/0xe83a7ce271e28f0b lrc: 0/0,0 mode: --/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0x557988ac98fe74f2 expref: -99 pid: 67035 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769075559.606634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075559.606637:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075559.606640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075559.912579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075559.912585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075559.912601:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075559.912610:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075559.912615:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075559.912626:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2545 00000100:00080000:1.0:1769075559.912638:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075559.912649:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075559.912695:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616406400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075559.912725:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075559.912729:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075559.912731:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075559.912738:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2545 00000100:00080000:1.0:1769075559.912742:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075559.912747:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075559.912757:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075559.912772:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616406528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075559.912792:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075559.912797:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075559.912800:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075559.912806:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2545 00000100:00080000:1.0:1769075559.912810:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075559.912814:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075559.912819:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075559.912834:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616406656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075559.912844:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075559.912847:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075559.912849:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075559.912853:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2545 00000100:00080000:1.0:1769075559.912857:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075559.912861:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075559.912866:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075559.912883:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616406784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075559.912902:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075559.912908:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075559.912912:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075559.912921:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2545 00000100:00080000:1.0:1769075559.912927:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075559.912935:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075559.912944:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075559.912970:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616406912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075559.912989:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075559.912998:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075559.913004:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075559.913016:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2545 00000100:00080000:1.0:1769075559.913026:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075559.913057:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075559.913065:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075559.913085:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616407040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075559.913097:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075559.913100:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075559.913102:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075559.913107:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2545 00000100:00080000:1.0:1769075559.913111:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075559.913122:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075559.913133:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075559.913153:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616407168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075560.616610:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075560.936597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075560.936632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075561.640590:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075561.960573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075561.960600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075561.960601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075562.664575:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075562.984560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075562.984563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075562.984564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075563.688689:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075564.008637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075564.008637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075564.008718:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075564.712695:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075565.032560:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075565.032561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075565.032565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075565.032574:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075565.032579:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075565.032589:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2550 00000100:00080000:1.0:1769075565.032599:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769075565.032602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075565.032609:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075565.032648:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616407296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075565.032673:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075565.032677:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075565.032679:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075565.032682:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2550 00000100:00080000:1.0:1769075565.032685:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075565.032688:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075565.032696:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075565.032707:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616407424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075565.032715:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075565.032719:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075565.032722:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075565.032727:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2550 00000100:00080000:1.0:1769075565.032731:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075565.032736:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075565.032740:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075565.032750:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616407552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075565.032757:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075565.032760:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075565.032762:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075565.032764:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2550 00000100:00080000:1.0:1769075565.032767:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075565.032769:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075565.032773:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075565.032781:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616407680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075565.032788:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075565.032791:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075565.032792:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075565.032796:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2550 00000100:00080000:1.0:1769075565.032798:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075565.032801:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075565.032804:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075565.032814:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616407808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075565.032823:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075565.032827:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075565.032828:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075565.032832:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2550 00000100:00080000:1.0:1769075565.032835:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075565.032838:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075565.032842:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075565.032853:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616407936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075565.032905:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075565.032910:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075565.032916:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075565.032931:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2550 00000100:00080000:1.0:1769075565.032942:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075565.032970:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075565.032980:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075565.033002:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616408064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075565.736575:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075566.056596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075566.056618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075566.056672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075566.760584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075567.080534:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075567.080543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075567.784604:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075568.104597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075568.104634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075568.104709:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075568.808577:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075569.128621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075569.128632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075569.128647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075569.832698:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075570.152593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075570.152606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075570.152695:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075570.152704:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075570.152711:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075570.152713:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075570.152726:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2555 00000100:00080000:3.0:1769075570.152758:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075570.152772:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075570.152832:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616408192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075570.152872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075570.152877:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075570.152880:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075570.152885:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2555 00000100:00080000:3.0:1769075570.152889:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075570.152895:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075570.152910:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075570.152928:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616408320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075570.152942:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075570.152953:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075570.152962:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075570.152983:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2555 00000100:00080000:3.0:1769075570.152987:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075570.152991:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075570.152998:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075570.153037:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616408448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075570.153049:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075570.153054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075570.153056:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075570.153061:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2555 00000100:00080000:3.0:1769075570.153069:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075570.153074:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075570.153100:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075570.153130:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616408576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075570.153148:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075570.153155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075570.153160:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075570.153191:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2555 00000100:00080000:3.0:1769075570.153215:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075570.153227:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075570.153247:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075570.153275:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616408704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075570.153330:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075570.153354:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075570.153357:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075570.153375:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2555 00000100:00080000:3.0:1769075570.153393:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075570.153402:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075570.153411:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075570.153434:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616408832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075570.153453:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075570.153459:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075570.153463:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075570.153512:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2555 00000100:00080000:3.0:1769075570.153528:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075570.153548:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075570.153569:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075570.153606:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616408960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075570.856572:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075571.176555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075571.176573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075571.176594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075571.880698:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075572.200536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075572.200543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075572.200564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075572.904827:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075573.224620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075573.224622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075573.928714:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075574.248542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075574.248542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075574.248635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075574.952712:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075575.272525:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075575.272565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075575.272578:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075575.272584:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075575.272588:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075575.272593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769075575.272598:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2560 00000100:00080000:0.0:1769075575.272610:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075575.272618:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075575.272657:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504380 x1855007616409088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075575.272685:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075575.272688:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075575.272690:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075575.272694:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2560 00000100:00080000:0.0:1769075575.272696:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075575.272699:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075575.272709:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075575.272721:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616409216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075575.272732:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075575.272738:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075575.272743:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075575.272753:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2560 00000100:00080000:0.0:1769075575.272755:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075575.272758:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075575.272763:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075575.272776:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616409344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075575.272785:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075575.272788:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075575.272790:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075575.272793:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2560 00000100:00080000:0.0:1769075575.272796:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075575.272800:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075575.272805:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075575.272817:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616409472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075575.272843:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075575.272847:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075575.272849:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075575.272855:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2560 00000100:00080000:0.0:1769075575.272859:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075575.272862:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075575.272866:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075575.272879:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616409600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075575.272902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075575.272909:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075575.272912:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075575.272939:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2560 00000100:00080000:0.0:1769075575.272945:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075575.272959:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075575.272976:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075575.272991:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616409728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075575.273038:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075575.273051:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075575.273054:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075575.273059:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2560 00000100:00080000:0.0:1769075575.273062:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075575.273065:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075575.273070:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075575.273083:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616409856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075575.976688:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075576.296590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075576.296593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075576.296602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075577.000749:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075577.320547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075577.320575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075577.320647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075577.320659:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075578.024656:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075578.344559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075578.344560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075578.344560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075579.048620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075579.368540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075579.368545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075579.368808:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075579.368841:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075580.072714:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075580.392562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075580.392571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769075580.392578:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075580.392587:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075580.392592:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075580.392602:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2566 00000100:00080000:0.0:1769075580.392615:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075580.392625:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075580.392626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075580.392674:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616409984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075580.392721:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075580.392726:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075580.392728:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075580.392733:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2566 00000100:00080000:0.0:1769075580.392736:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075580.392741:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075580.392751:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075580.392773:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616409984:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075580.392774:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616410112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075580.392792:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075580.392796:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075580.392797:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075580.392801:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2566 00000100:00080000:0.0:1769075580.392805:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075580.392808:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075580.392812:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075580.392823:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616410240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075580.392838:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075580.392841:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075580.392843:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075580.392846:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2566 00000100:00080000:0.0:1769075580.392852:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075580.392855:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075580.392859:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769075580.392867:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075580.392879:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616410368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075580.392890:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075580.392893:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075580.392895:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075580.392898:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2566 00000100:00080000:0.0:1769075580.392902:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075580.392905:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075580.392909:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075580.392923:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616410496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075580.392933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075580.392936:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075580.392938:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075580.392942:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2566 00000100:00080000:0.0:1769075580.392946:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769075580.392947:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075580/real 1769075580] req@ffff995b83504000 x1855007616409984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075580.392951:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075580.392958:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075580.392971:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616410624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075580.392980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075580.392986:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769075580.392987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616409984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075580.392988:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075580.392999:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2566 00000100:00080000:0.0:1769075580.393005:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075580.393012:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769075580.393013:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769075580.393017:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769075580.393021:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075580.393028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616409984:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075580.393031:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504380 x1855007616410752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075580.393062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616410112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075580.393089:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075580.393102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075580/real 1769075580] req@ffff995b83507100 x1855007616410112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075580.393145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616410112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075580.393155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075580.393158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075580.393161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616410112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075580.393167:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616410240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075580.393188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075580.393198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075580/real 1769075580] req@ffff995b83506300 x1855007616410240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075580.393209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616410240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075580.393218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075580.393221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075580.393233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616410240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075580.393239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616410368:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075580.393255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075580.393264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075580/real 1769075580] req@ffff995b83507800 x1855007616410368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075580.393275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616410368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075580.393282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075580.393285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075580.393288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616410368:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075580.393294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616410496:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075580.393306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075580.393314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075580/real 1769075580] req@ffff995b83504e00 x1855007616410496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075580.393323:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616410496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075580.393332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075580.393334:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075580.393337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616410496:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075580.393342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616410624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075580.393355:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075580.393363:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075580/real 1769075580] req@ffff995b83505180 x1855007616410624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075580.393372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616410624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075580.393379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075580.393382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075580.393384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616410624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075580.393389:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616410752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075580.393401:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075580.393409:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075580/real 1769075580] req@ffff995b83504380 x1855007616410752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075580.393419:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504380 x1855007616410752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075580.393426:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075580.393428:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075580.393431:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616410752:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769075581.096696:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075581.416644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075581.416747:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075582.120648:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075582.440558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075582.440565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075582.440589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075583.144666:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075583.464574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075583.464578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075583.464599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075584.168565:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075584.488641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075584.488694:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075584.488698:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075585.192603:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075585.512610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075585.512615:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075585.512626:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075585.512632:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075585.512643:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2571 00000100:00100000:0.0:1769075585.512649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075585.512654:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075585.512665:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075585.512693:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075585.512703:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616410880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075585.512733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075585.512737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075585.512739:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075585.512743:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2571 00000100:00080000:1.0:1769075585.512746:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075585.512750:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075585.512759:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075585.512770:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616411008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075585.512781:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075585.512783:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075585.512784:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075585.512788:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2571 00000100:00080000:1.0:1769075585.512790:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075585.512792:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075585.512796:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075585.512805:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616411136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075585.512815:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075585.512820:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075585.512821:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075585.512825:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2571 00000100:00080000:1.0:1769075585.512830:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075585.512839:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075585.512846:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075585.512861:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616411264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075585.512872:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075585.512880:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075585.512884:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075585.512892:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2571 00000100:00080000:1.0:1769075585.512896:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075585.512903:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075585.512912:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075585.512923:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616411392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075585.512940:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075585.512943:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075585.512945:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075585.512948:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2571 00000100:00080000:1.0:1769075585.512951:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075585.512953:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075585.512956:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075585.512966:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616411520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075585.512974:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075585.512977:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075585.512979:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075585.512985:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2571 00000100:00080000:1.0:1769075585.512992:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075585.512997:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075585.513005:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075585.513040:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616411648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075586.216565:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075586.536540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075586.536582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075586.536603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075586.536637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075587.240630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075587.560549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075587.560549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075587.560602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075588.264659:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075588.584600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075588.584688:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075588.584716:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075589.288579:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075589.608556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075589.608557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075589.608594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075590.312642:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075590.632546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075590.632546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075590.632547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075590.632576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075590.632727:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616411776:192.168.204.132@tcp:250:kworker.0 00000800:00000100:3.0:1769075590.632792:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075590.632830:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075590/real 1769075590] req@ffff995b85ac9880 x1855007616411776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075590.632858:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616411776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075590.632873:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075590.632879:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075590.632888:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616411776:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075590.632914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616411904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075590.632935:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075590.632966:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075590/real 1769075590] req@ffff995b85aca300 x1855007616411904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075590.632976:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616411904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075590.632982:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075590.632984:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075590.632986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616411904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075590.632991:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075590.633004:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075590.633028:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075590/real 1769075590] req@ffff995b85acb800 x1855007616412032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075590.633036:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616412032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075590.633044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075590.633046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075590.633048:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075590.633052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075590.633065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075590.633072:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075590/real 1769075590] req@ffff995b85ac8a80 x1855007616412160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075590.633080:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616412160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075590.633087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075590.633089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075590.633091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075590.633095:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412288:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075590.633105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075590.633112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075590/real 1769075590] req@ffff995b85acbb80 x1855007616412288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075590.633121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616412288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075590.633127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075590.633129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075590.633131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075590.633135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616412416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075590.633145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075590.633152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075590/real 1769075590] req@ffff995b85ac9c00 x1855007616412416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075590.633160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616412416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075590.633166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075590.633169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075590.633171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616412416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075590.633186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616412544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075590.633197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075590.633204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075590/real 1769075590] req@ffff995b85ac8380 x1855007616412544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075590.633213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616412544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075590.633219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075590.633221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075590.633223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616412544:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075591.336600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075591.656602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075591.656603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075591.656693:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075592.360754:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075592.680520:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075592.680566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075592.680569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075593.384592:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075593.704552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075593.704553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075593.704557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075594.408651:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075594.728581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075594.728605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075594.728606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075595.432670:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075595.752552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075595.752559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075595.752607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075595.752771:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616412672:192.168.204.132@tcp:250:kworker.0 00000800:00000100:0.0:1769075595.752931:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075595.752979:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075595/real 1769075595] req@ffff995b85acaa00 x1855007616412672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075595.753007:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616412672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075595.753021:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075595.753027:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075595.753037:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616412672:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075595.753069:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075595.753096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075595.753142:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075595/real 1769075595] req@ffff995b85acb100 x1855007616412800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075595.753172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616412800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075595.753183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075595.753188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075595.753192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075595.753199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075595.753241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075595.753254:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075595/real 1769075595] req@ffff995b85ac8380 x1855007616412928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075595.753268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616412928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075595.753281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075595.753288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075595.753294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616412928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075595.753305:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413056:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075595.753343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075595.753364:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075595/real 1769075595] req@ffff995b85ac9c00 x1855007616413056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075595.753402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616413056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075595.753421:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075595.753425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075595.753428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075595.753433:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413184:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075595.753451:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075595.753462:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075595/real 1769075595] req@ffff995b85acbb80 x1855007616413184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075595.753497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616413184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075595.753505:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075595.753508:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075595.753512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075595.753545:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616413312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075595.753560:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075595.753568:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075595/real 1769075595] req@ffff995b85ac8a80 x1855007616413312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075595.753576:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616413312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075595.753582:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075595.753584:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075595.753587:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616413312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075595.753591:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616413440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075595.753601:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075595.753607:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075595/real 1769075595] req@ffff995b85acb800 x1855007616413440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075595.753614:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616413440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075595.753620:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075595.753622:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075595.753624:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616413440:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075596.456678:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075596.776563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075596.776563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075596.776609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075597.480653:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075597.800536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075597.800559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075598.504593:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075598.824529:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075598.824544:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075598.824571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075599.528745:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075599.848630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075599.848635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075599.848636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075600.552675:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075600.872617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075600.872624:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075600.872656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075600.872739:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075600.872747:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075600.872753:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075600.872767:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2586 00000100:00080000:1.0:1769075600.872804:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075600.872815:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075600.872874:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616413568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075600.872906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075600.872910:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075600.872912:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075600.872916:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2586 00000100:00080000:1.0:1769075600.872920:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075600.872924:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075600.872937:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075600.872953:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616413696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075600.872962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616413568:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075600.872972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075600.872977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075600.872978:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075600.872982:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2586 00000100:00080000:1.0:1769075600.872985:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075600.872990:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075600.872995:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075600.873008:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616413824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075600.873016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075600.873019:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075600.873021:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075600.873028:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2586 00000100:00080000:1.0:1769075600.873041:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075600.873045:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075600.873051:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075600.873062:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616413952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075600.873070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075600.873073:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075600.873075:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075600.873085:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2586 00000100:00080000:1.0:1769075600.873088:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075600.873091:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075600.873095:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075600.873105:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616414080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075600.873125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413696:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075600.873158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075600.873162:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075600.873164:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075600.873173:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2586 00000100:00080000:1.0:1769075600.873201:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075600.873205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413824:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075600.873214:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075600.873220:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075600.873236:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616414208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075600.873244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413952:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769075600.873250:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075600.873254:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075600.873257:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075600.873265:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2586 00000100:00080000:1.0:1769075600.873271:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075600.873276:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075600.873283:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075600.873286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075600.873309:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616414336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075600.873332:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616414208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075600.873368:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616414336:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769075601.576661:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075601.896644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075601.896657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075601.896825:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075602.600672:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075602.920578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075602.920616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075602.920636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075602.920643:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075603.624650:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075603.944589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075603.944648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075603.944659:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:0.0:1769075603.945076:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:0.0:1769075603.945133:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769075603.945157:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075603.945203:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075603.945211:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075603.945219:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075603.945225:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075603.945232:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075603.945239:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769075603.945295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075600/real 1769075603] req@ffff995b868c7480 x1855007616414336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075603.945326:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616414336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075603.945343:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075603.945350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075603.945356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616414336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075603.945366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075600/real 1769075603] req@ffff995b868c4700 x1855007616414080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075603.945386:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616414080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075603.945415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075603.945425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075603.945429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075603.945439:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075600/real 1769075603] req@ffff995b868c6a00 x1855007616414208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075603.945487:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616414208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075603.945498:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075603.945504:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075603.945510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616414208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075603.945522:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075600/real 1769075603] req@ffff995b868c5500 x1855007616413696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075603.945546:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616413696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075603.945558:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075603.945565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075603.945570:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075603.945579:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075600/real 1769075603] req@ffff995b868c7800 x1855007616413824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075603.945594:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616413824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075603.945610:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075603.945618:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075603.945622:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075603.945634:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075600/real 1769075603] req@ffff995b868c4a80 x1855007616413952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075603.945661:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616413952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075603.945673:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075603.945680:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075603.945690:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616413952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075603.945702:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075600/real 1769075603] req@ffff995b868c5c00 x1855007616413568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075603.945735:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616413568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075603.945744:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075603.945749:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075603.945753:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616413568:192.168.204.132@tcp:250:kworker.0 00010000:00010000:1.0:1769075604.648764:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075604.968624:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075604.968638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075604.968766:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075605.672752:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075605.992570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075605.992571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075605.992572:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075605.992596:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075605.992602:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075605.992614:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2591 00000100:00080000:1.0:1769075605.992628:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769075605.992638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075605.992644:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075605.992729:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616414464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075605.992795:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075605.992800:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075605.992803:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075605.992810:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2591 00000100:00080000:1.0:1769075605.992816:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075605.992822:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075605.992836:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075605.992854:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616414592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075605.992868:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075605.992872:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075605.992874:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075605.992878:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2591 00000100:00080000:1.0:1769075605.992882:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075605.992888:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075605.992893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616414464:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769075605.992897:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075605.992920:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616414720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075605.992933:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075605.992938:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075605.992940:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075605.992946:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2591 00000100:00080000:1.0:1769075605.992952:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075605.992958:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075605.992966:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769075605.992988:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075605.992996:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616414848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075605.993011:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075605.993015:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075605.993018:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075605.993022:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2591 00000100:00080000:1.0:1769075605.993028:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075605.993033:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075605.993041:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075605.993047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075605/real 1769075605] req@ffff995b868c6d80 x1855007616414464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075605.993063:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616414976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075605.993081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616414464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075605.993092:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075605.993102:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075605.993107:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075605.993107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075605.993114:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2591 00000100:00080000:1.0:1769075605.993119:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075605.993127:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769075605.993133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769075605.993139:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075605.993143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616414464:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075605.993159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075605.993165:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616415104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769075605.993200:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075605.993215:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075605.993220:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075605.993223:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075605.993232:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2591 00000100:00100000:3.0:1769075605.993236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075605/real 1769075605] req@ffff995b868c6300 x1855007616414592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075605.993239:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075605.993252:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075605.993264:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075605.993283:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616415232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075605.993296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616414592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075605.993312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075605.993320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075605.993325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075605.993363:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075605.993392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075605.993408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075605/real 1769075605] req@ffff995b868c7100 x1855007616414720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075605.993424:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616414720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075605.993438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075605.993443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075605.993447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075605.993902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075605.993941:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075605.993960:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075605/real 1769075605] req@ffff995b868c5c00 x1855007616414848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075605.993979:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616414848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075605.993989:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075605.993993:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075605.993999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075605.994008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075605.994054:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075605.994071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075605/real 1769075605] req@ffff995b868c4a80 x1855007616414976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075605.994090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616414976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075605.994120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075605.994124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075605.994128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616414976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075605.994145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616415104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075605.994174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075605.994191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075605/real 1769075605] req@ffff995b868c7800 x1855007616415104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075605.994209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616415104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075605.994220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075605.994224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075605.994229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616415104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075605.994237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616415232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075605.994264:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075605.994280:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075605/real 1769075605] req@ffff995b868c5500 x1855007616415232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075605.994299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616415232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075605.994311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075605.994324:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075605.994329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616415232:192.168.204.132@tcp:38:kworker.0 00010000:00010000:3.0:1769075606.696600:0:52323:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1769075606.696665:0:13582:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1769075606.696687:0:13582:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff995b87ecd000 lock: ffff995b852fee00/0xe83a7ce271e2205b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x557988ac98fdb7eb expref: -99 pid: 65253 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1769075606.696697:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075606.696747:0:13582:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff995b87ecd000 lock: ffff995b852fee00/0xe83a7ce271e2205b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x557988ac98fdb7eb expref: -99 pid: 65253 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1769075606.696754:0:13582:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1769075606.696782:0:13582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff995b87ecd000 lock: ffff995b852fee00/0xe83a7ce271e2205b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x557988ac98fdb7eb expref: -99 pid: 65253 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1769075606.696802:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075606.696804:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075606.696806:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075606.696810:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff995b83504380 x1855007616415360/t0(0) o103->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075607.016575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075607.016761:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075607.016765:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075607.016847:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075607.720694:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075608.040558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075608.040558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075608.040598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075608.744631:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075609.064577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075609.064577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075609.064577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075609.768676:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075610.088560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075610.088560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075610.088642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075610.088666:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075610.792701:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075611.112651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075611.112702:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075611.112703:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075611.112704:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075611.112835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616415488:192.168.204.132@tcp:250:kworker.0 00000800:00000100:3.0:1769075611.112970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075611.113106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075611/real 1769075611] req@ffff995b85aca300 x1855007616415488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075611.113190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616415488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075611.113218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075611.113226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075611.113233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616415488:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075611.113275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616415616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075611.113315:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075611.113352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075611/real 1769075611] req@ffff995b85ac9880 x1855007616415616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075611.113377:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616415616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075611.113392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075611.113399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075611.113406:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616415616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075611.113419:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616415744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075611.113455:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075611.113515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075611/real 1769075611] req@ffff995b85acb800 x1855007616415744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075611.113540:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616415744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075611.113560:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075611.113567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075611.113573:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616415744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075611.113585:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616415872:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075611.113668:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075611.113689:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075611/real 1769075611] req@ffff995b85ac8a80 x1855007616415872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075611.113767:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616415872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075611.113785:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075611.113792:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075611.113798:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616415872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075611.113810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416000:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769075611.113849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075611.113870:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075611/real 1769075611] req@ffff995b85acbb80 x1855007616416000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075611.113893:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616416000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075611.113911:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075611.113917:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075611.113924:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075611.113937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616416128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075611.113971:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075611.113990:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075611/real 1769075611] req@ffff995b85ac9c00 x1855007616416128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075611.114027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616416128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075611.114049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075611.114053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075611.114058:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616416128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075611.114068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616416256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769075611.114098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075611.114114:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075611/real 1769075611] req@ffff995b85ac8380 x1855007616416256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075611.114131:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616416256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075611.114143:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075611.114148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075611.114153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616416256:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075611.816781:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075612.136597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075612.136605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075612.136630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075612.840704:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075613.160529:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075613.160536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075613.160597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075613.864673:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075614.184509:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075614.184514:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075614.184537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075614.184589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075614.888599:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075615.208613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075615.208616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075615.208638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075615.912651:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075616.232553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075616.232626:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075616.232633:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075616.232637:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075616.232654:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075616.232654:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2601 00000100:00080000:3.0:1769075616.232672:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075616.232682:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075616.232724:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616416384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075616.232748:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075616.232752:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075616.232753:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075616.232757:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2601 00000100:00080000:3.0:1769075616.232759:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075616.232762:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075616.232773:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075616.232784:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616416512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075616.232793:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075616.232796:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075616.232797:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075616.232801:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2601 00000100:00080000:3.0:1769075616.232804:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075616.232808:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075616.232810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616416384:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769075616.232812:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075616.232830:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616416640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075616.232837:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075616.232840:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075616.232843:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075616.232846:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2601 00000100:00080000:3.0:1769075616.232849:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075616.232852:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075616.232861:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075616.232871:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616416768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075616.232877:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075616.232880:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075616.232881:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075616.232884:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2601 00000100:00080000:3.0:1769075616.232887:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075616.232890:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075616.232894:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769075616.232903:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075616.232907:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616416896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075616.232917:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075616.232920:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075616.232922:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075616.232927:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2601 00000100:00080000:3.0:1769075616.232930:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075616.232933:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075616.232938:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075616.232949:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616417024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075616.232958:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075616/real 1769075616] req@ffff995b8455b480 x1855007616416384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075616.232960:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075616.232970:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075616.232976:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075616.232990:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2601 00000100:00080000:3.0:1769075616.232995:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075616.232999:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075616.233002:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075616.233009:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616416384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075616.233014:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616417152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075616.233054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075616.233061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075616.233065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616416384:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075616.233106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075616.233128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075616.233138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075616/real 1769075616] req@ffff995b84558000 x1855007616416512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075616.233148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616416512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075616.233154:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075616.233156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075616.233159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075616.233163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075616.233175:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075616.233181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075616/real 1769075616] req@ffff995b8455a680 x1855007616416640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075616.233198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616416640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075616.233213:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075616.233217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075616.233221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075616.233229:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416768:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075616.233251:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075616.233261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075616/real 1769075616] req@ffff995b84558a80 x1855007616416768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075616.233269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616416768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075616.233276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075616.233278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075616.233280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075616.233285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075616.233307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075616.233314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075616/real 1769075616] req@ffff995b84559500 x1855007616416896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075616.233322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616416896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075616.233328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075616.233330:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075616.233332:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616416896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075616.233336:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616417024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075616.233346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075616.233352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075616/real 1769075616] req@ffff995b8455aa00 x1855007616417024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075616.233359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616417024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075616.233369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075616.233378:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075616.233380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616417024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075616.233384:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616417152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075616.233395:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075616.233401:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075616/real 1769075616] req@ffff995b8455bb80 x1855007616417152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075616.233409:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616417152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075616.233415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075616.233417:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075616.233419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616417152:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769075616.936632:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075617.256528:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075617.256566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075617.256617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075617.960592:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075618.280579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075618.280581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075618.280584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075618.984629:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075619.304619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075619.304622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075619.304703:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075619.304718:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075620.008658:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075620.328537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075620.328540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075620.328562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075620.328594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075621.032596:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075621.352556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075621.352556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075621.352580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075621.352611:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075621.352619:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075621.352624:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075621.352632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075621.352635:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2607 00000100:00080000:1.0:1769075621.352648:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075621.352657:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075621.352695:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616417280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075621.352730:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075621.352736:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075621.352739:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075621.352745:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2607 00000100:00080000:1.0:1769075621.352750:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075621.352756:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616417280:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075621.352757:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075621.352772:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075621.352789:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616417408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075621.352802:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075621.352806:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075621.352809:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769075621.352815:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075621.352816:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2607 00000100:00080000:1.0:1769075621.352824:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075621.352829:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075621.352835:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075621.352848:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616417536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075621.352860:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075621.352863:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075621/real 1769075621] req@ffff995b868c6a00 x1855007616417280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075621.352866:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075621.352871:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075621.352881:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2607 00000100:00080000:1.0:1769075621.352889:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075621.352894:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616417280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075621.352898:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769075621.352909:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769075621.352910:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769075621.352916:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075621.352919:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616417280:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075621.352928:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616417664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075621.352940:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616417408:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075621.352942:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075621.352948:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075621.352951:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075621.352957:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2607 00000800:00000100:0.0:1769075621.352958:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075621.352967:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075621.352974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075621/real 1769075621] req@ffff995b868c4700 x1855007616417408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075621.352975:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075621.352985:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075621.352990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616417408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075621.352997:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075621.353004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075621.353005:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616417792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075621.353008:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616417408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075621.353023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616417536:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075621.353031:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075621.353036:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1769075621.353037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075621.353041:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075621.353049:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2607 00000100:00100000:0.0:1769075621.353054:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075621/real 1769075621] req@ffff995b868c7480 x1855007616417536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075621.353056:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075621.353065:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075621.353074:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075621.353080:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616417536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075621.353090:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616417920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075621.353112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075621.353116:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075621.353117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075621.353124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616417536:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075621.353126:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075621.353130:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075621.353136:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2607 00000100:00100000:0.0:1769075621.353142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616417664:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769075621.353142:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075621.353151:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075621.353159:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769075621.353166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075621.353176:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616418048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075621.353178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075621/real 1769075621] req@ffff995b868c5500 x1855007616417664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075621.353206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616417664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075621.353212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075621.353218:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075621.353231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616417664:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075621.353237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616417792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075621.353249:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075621.353256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075621/real 1769075621] req@ffff995b868c7800 x1855007616417792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075621.353263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616417792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075621.353269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075621.353271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075621.353274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616417792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075621.353278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616417920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075621.353287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075621.353293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075621/real 1769075621] req@ffff995b868c4a80 x1855007616417920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075621.353300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616417920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075621.353311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075621.353313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075621.353315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616417920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075621.353325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616418048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075621.353336:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075621.353342:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075621/real 1769075621] req@ffff995b868c5c00 x1855007616418048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075621.353350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616418048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075621.353355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075621.353357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075621.353359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616418048:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075622.056641:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075622.376570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075622.376571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075622.376615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075623.080577:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075623.400546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075623.400577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075624.104581:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075624.424601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075624.424657:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075624.424660:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075625.128628:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075625.448549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075625.448593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075625.448620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075626.152654:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075626.472590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075626.472646:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075626.472647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075626.472657:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075626.472663:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075626.472675:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2612 00000100:00080000:3.0:1769075626.472690:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075626.472711:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075626.472749:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616418176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075626.472774:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075626.472778:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075626.472780:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075626.472785:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2612 00000100:00080000:3.0:1769075626.472789:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075626.472794:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075626.472805:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616418176:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769075626.472806:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075626.472824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616418304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075626.472838:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075626.472841:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075626.472844:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075626.472848:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2612 00000100:00080000:3.0:1769075626.472852:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075626.472856:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075626.472862:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769075626.472867:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075626.472877:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616418432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075626.472888:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075626.472891:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075626.472893:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075626.472898:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2612 00000100:00080000:3.0:1769075626.472902:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075626.472907:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075626/real 1769075626] req@ffff995b8455bb80 x1855007616418176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075626.472908:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075626.472916:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075626.472927:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616418176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075626.472937:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769075626.472941:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616418560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075626.472944:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075626.472951:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616418176:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075626.472962:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075626.472966:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075626.472968:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075626.472973:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2612 00000100:00080000:3.0:1769075626.472977:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075626.472982:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075626.472986:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616418304:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769075626.472988:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075626.473005:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616418688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075626.473013:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075626.473023:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075626.473027:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075626.473030:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075626.473035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075626/real 1769075626] req@ffff995b8455aa00 x1855007616418304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075626.473037:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2612 00000100:00080000:3.0:1769075626.473042:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075626.473049:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075626.473057:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075626.473064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616418304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075626.473073:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616418816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075626.473077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075626.473086:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075626.473092:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075626.473093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616418304:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075626.473100:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075626.473103:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075626.473104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616418432:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075626.473110:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2612 00000100:00080000:3.0:1769075626.473115:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075626.473119:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075626.473124:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769075626.473134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075626.473136:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616418944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075626.473151:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075626/real 1769075626] req@ffff995b84559500 x1855007616418432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075626.473169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616418432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075626.473180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075626.473182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075626.473185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616418432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075626.473189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616418560:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075626.473214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075626.473222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075626/real 1769075626] req@ffff995b84558a80 x1855007616418560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075626.473230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616418560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075626.473236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075626.473238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075626.473241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616418560:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075626.473255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616418688:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075626.473264:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075626.473271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075626/real 1769075626] req@ffff995b8455a680 x1855007616418688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075626.473279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616418688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075626.473285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075626.473286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075626.473288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616418688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075626.473292:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616418816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075626.473301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075626.473307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075626/real 1769075626] req@ffff995b84558000 x1855007616418816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075626.473314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616418816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075626.473320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075626.473321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075626.473323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616418816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075626.473326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616418944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075626.473335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075626.473367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075626/real 1769075626] req@ffff995b8455b480 x1855007616418944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075626.473374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616418944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075626.473380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075626.473382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075626.473384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616418944:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769075627.176598:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075627.496594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075627.496601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075627.496628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075627.496663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075628.200578:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075628.520560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075628.520560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075628.520561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075628.520623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075629.224605:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075629.544573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075629.544581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075629.544589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075630.248582:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075630.568563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075630.568565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075630.568568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075630.568624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075631.272680:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075631.592580:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075631.592581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075631.592591:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075631.592597:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075631.592607:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2617 00000100:00080000:3.0:1769075631.592620:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075631.592626:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075631.592639:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075631.592665:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616419072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075631.592695:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075631.592701:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075631.592704:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075631.592715:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2617 00000100:00080000:3.0:1769075631.592722:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075631.592728:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616419072:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075631.592736:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075631.592747:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075631.592762:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616419200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075631.592771:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075631.592774:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075631.592775:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075631.592778:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2617 00000100:00080000:3.0:1769075631.592784:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075631.592788:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075631.592794:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075631.592811:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616419328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075631.592821:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075631.592822:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075631.592835:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075631.592838:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075631.592845:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2617 00000100:00080000:3.0:1769075631.592852:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075631.592858:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075631.592864:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075631.592881:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616419456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075631.592891:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075631.592893:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075631/real 1769075631] req@ffff995b84559880 x1855007616419072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075631.592898:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075631.592903:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075631.592922:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2617 00000100:00100000:0.0:1769075631.592933:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616419072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075631.592944:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075631.592948:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075631.592951:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616419072:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075631.592976:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616419200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075631.593000:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075631.593001:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075631.593008:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075631.593014:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075631.593016:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075631/real 1769075631] req@ffff995b84558700 x1855007616419200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075631.593031:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616419200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075631.593037:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075631.593039:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075631.593041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616419200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075631.593045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616419328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075631.593058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075631.593066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075631/real 1769075631] req@ffff995b8455a300 x1855007616419328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075631.593073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616419328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075631.593080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075631.593083:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075631.593086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616419328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075631.593090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616419456:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075631.593114:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075631.593126:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075631/real 1769075631] req@ffff995b8455b100 x1855007616419456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075631.593141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616419456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075631.593146:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616419584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075631.593203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075631.593212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075631.593219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616419456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075631.593282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616419584:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769075631.593292:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075631.593296:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075631.593299:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075631.593304:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2617 00000100:00080000:3.0:1769075631.593310:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769075631.593317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075631.593323:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075631.593330:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075631.593337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075631/real 1769075631] req@ffff995b84559c00 x1855007616419584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075631.593346:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616419712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075631.593351:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616419584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075631.593425:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075631.593429:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075631.593430:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075631.593436:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075631.593443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075631.593448:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2617 00000100:00080000:3.0:1769075631.593476:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075631.593480:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075631.593484:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616419584:192.168.204.132@tcp:8:kworker.0 00000008:00100000:3.0:1769075631.593486:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075631.593509:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616419840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075631.593523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616419712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075631.593553:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075631.593568:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075631/real 1769075631] req@ffff995b8455ad80 x1855007616419712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075631.593582:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616419712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075631.593591:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075631.593595:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075631.593600:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616419712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075631.593614:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616419840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075631.593633:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075631.593645:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075631/real 1769075631] req@ffff995b8455b100 x1855007616419840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075631.593658:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616419840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075631.593667:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075631.593671:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075631.593675:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616419840:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075632.296633:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075632.616584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075632.616605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075632.616624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075633.320597:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075633.640607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075633.640629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075633.640634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075634.344641:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075634.664593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075634.664642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075634.664649:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075635.368608:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075635.688641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075635.688689:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075635.688695:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075636.392598:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075636.712565:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075636.712570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075636.712570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075636.712575:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075636.712581:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075636.712595:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2622 00000100:00080000:3.0:1769075636.712609:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075636.712620:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075636.712661:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616419968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075636.712691:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075636.712694:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075636.712696:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075636.712700:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2622 00000100:00080000:3.0:1769075636.712703:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075636.712707:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075636.712715:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075636.712729:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616420096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075636.712730:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616419968:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075636.712744:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075636.712746:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075636.712747:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075636.712750:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2622 00000100:00080000:3.0:1769075636.712753:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075636.712756:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075636.712760:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075636.712768:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616420224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075636.712775:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075636.712778:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075636.712779:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075636.712781:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2622 00000100:00080000:3.0:1769075636.712783:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075636.712786:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075636.712789:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769075636.712798:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075636.712799:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616420352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075636.712808:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075636.712811:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075636.712812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075636.712815:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2622 00000100:00080000:3.0:1769075636.712818:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075636.712820:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075636.712823:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075636.712831:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616420480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075636.712837:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075636.712840:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075636/real 1769075636] req@ffff995b8455a300 x1855007616419968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075636.712840:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075636.712844:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075636.712852:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2622 00000100:00080000:3.0:1769075636.712858:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075636.712861:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075636.712864:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075636.712866:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616419968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075636.712874:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616420608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075636.712901:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075636.712902:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075636.712908:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075636.712911:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075636.712912:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075636.712918:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616419968:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075636.712919:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2622 00000100:00080000:3.0:1769075636.712926:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075636.712930:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075636.712933:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075636.712941:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616420736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075636.712945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075636.712966:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075636.712976:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075636/real 1769075636] req@ffff995b84558700 x1855007616420096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075636.712984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616420096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075636.712990:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075636.712992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075636.712995:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075636.712998:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075636.713007:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075636.713013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075636/real 1769075636] req@ffff995b84559880 x1855007616420224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075636.713020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616420224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075636.713027:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075636.713029:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075636.713030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075636.713034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420352:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075636.713044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075636.713050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075636/real 1769075636] req@ffff995b8455b480 x1855007616420352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075636.713057:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616420352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075636.713063:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075636.713065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075636.713066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075636.713070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420480:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075636.713078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075636.713084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075636/real 1769075636] req@ffff995b84558000 x1855007616420480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075636.713090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616420480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075636.713096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075636.713098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075636.713099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075636.713103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616420608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075636.713116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075636.713126:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075636/real 1769075636] req@ffff995b8455a680 x1855007616420608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075636.713138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616420608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075636.713148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075636.713151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075636.713154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616420608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075636.713203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616420736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075636.713219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075636.713227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075636/real 1769075636] req@ffff995b84558a80 x1855007616420736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075636.713235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616420736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075636.713241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075636.713243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075636.713245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616420736:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769075637.416595:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075637.736518:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075637.736560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075637.736627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075637.736662:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075638.440616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075638.760566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075638.760567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075638.760653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075639.464608:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075639.784559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075639.784560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075639.784587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075639.784621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075640.488594:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075640.808546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075640.808547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075640.808553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075641.512586:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075641.832556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075641.832556:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075641.832557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075641.832567:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075641.832572:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075641.832581:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2627 00000100:00080000:3.0:1769075641.832591:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075641.832598:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075641.832631:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616420864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075641.832653:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075641.832656:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075641.832657:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075641.832660:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2627 00000100:00080000:3.0:1769075641.832662:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075641.832666:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075641.832674:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075641.832684:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616420864:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075641.832686:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616420992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075641.832700:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075641.832702:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075641.832703:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075641.832713:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2627 00000100:00080000:3.0:1769075641.832717:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075641.832721:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075641.832726:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075641.832738:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616421120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075641.832749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075641.832750:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075641.832756:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075641.832759:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075641.832764:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2627 00000100:00080000:3.0:1769075641.832769:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075641.832773:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075641.832778:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075641.832785:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075641/real 1769075641] req@ffff995b84559500 x1855007616420864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075641.832791:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616421248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075641.832803:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075641.832807:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616420864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075641.832808:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075641.832812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075641.832819:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2627 00000100:00080000:0.0:1769075641.832825:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075641.832825:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075641.832829:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769075641.832833:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769075641.832835:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075641.832839:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616420864:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075641.832847:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616421376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075641.832855:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075641.832857:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075641.832858:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075641.832861:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2627 00000100:00100000:0.0:1769075641.832865:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420992:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075641.832866:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075641.832872:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075641.832877:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075641.832886:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616421504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075641.832892:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075641.832903:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075641.832906:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075641.832907:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075641.832909:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075641/real 1769075641] req@ffff995b8455aa00 x1855007616420992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075641.832912:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2627 00000100:00080000:3.0:1769075641.832918:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075641.832925:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075641.832929:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075641.832930:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616420992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075641.832940:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769075641.832941:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616421632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075641.832945:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075641.832952:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616420992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075641.832961:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616421120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075641.832977:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075641.832984:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075641/real 1769075641] req@ffff995b8455bb80 x1855007616421120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075641.832996:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616421120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075641.833002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075641.833004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075641.833006:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616421120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075641.833010:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616421248:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075641.833020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075641.833026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075641/real 1769075641] req@ffff995b84558a80 x1855007616421248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075641.833033:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616421248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075641.833039:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075641.833041:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075641.833043:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616421248:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075641.833046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616421376:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075641.833054:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075641.833060:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075641/real 1769075641] req@ffff995b8455a680 x1855007616421376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075641.833067:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616421376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075641.833077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075641.833079:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075641.833113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616421376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075641.833131:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616421504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075641.833147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075641.833155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075641/real 1769075641] req@ffff995b84558000 x1855007616421504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075641.833162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616421504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075641.833168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075641.833170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075641.833172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616421504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075641.833176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616421632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075641.833186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075641.833192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075641/real 1769075641] req@ffff995b8455b480 x1855007616421632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075641.833199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616421632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075641.833204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075641.833205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075641.833208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616421632:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075642.536573:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075642.856588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075642.856621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075642.856665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075643.560633:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075643.880621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075643.880628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075643.880638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075644.584604:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075644.904550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075644.904611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075644.904613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075645.608621:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075645.928562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075645.928562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075645.928600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075646.632609:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075646.952539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075646.952566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075646.952612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075646.952724:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616421760:192.168.204.132@tcp:250:kworker.0 00000800:00000100:0.0:1769075646.952785:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075646.952823:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075646/real 1769075646] req@ffff995b85acb100 x1855007616421760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075646.952852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616421760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075646.952866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075646.952872:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075646.952878:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616421760:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075646.952907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616421888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075646.952929:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075646.952940:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075646/real 1769075646] req@ffff995b85acaa00 x1855007616421888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075646.952967:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616421888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075646.952984:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075646.953019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075646.953029:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616421888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075646.953041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075646.953076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075646.953090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075646/real 1769075646] req@ffff995b85ac8380 x1855007616422016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075646.953105:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616422016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075646.953127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075646.953132:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075646.953137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075646.953145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422144:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075646.953167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075646.953177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075646/real 1769075646] req@ffff995b85ac9c00 x1855007616422144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075646.953185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616422144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075646.953191:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075646.953194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075646.953196:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422144:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075646.953208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075646.953220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075646.953226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075646/real 1769075646] req@ffff995b85acbb80 x1855007616422272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075646.953234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616422272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075646.953239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075646.953241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075646.953243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075646.953246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616422400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075646.953255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075646.953261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075646/real 1769075646] req@ffff995b85ac8a80 x1855007616422400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075646.953269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616422400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075646.953274:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075646.953276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075646.953278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616422400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075646.953281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616422528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075646.953289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075646.953294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075646/real 1769075646] req@ffff995b85acb800 x1855007616422528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075646.953301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616422528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075646.953310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075646.953312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075646.953315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616422528:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769075647.656602:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075647.976572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075647.976589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075647.976665:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075648.680654:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075649.000545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075649.000545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075649.000582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075649.704594:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075650.024565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075650.024565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075650.024566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075650.728621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075651.048540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075651.048567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075651.048568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075651.752620:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075652.072545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075652.072551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075652.072582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075652.072614:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075652.072621:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075652.072625:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075652.072637:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2637 00000100:00080000:3.0:1769075652.072648:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075652.072656:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075652.072691:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616422656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075652.072714:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075652.072717:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075652.072719:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075652.072722:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2637 00000100:00080000:3.0:1769075652.072726:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075652.072731:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075652.072740:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075652.072754:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616422656:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075652.072756:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616422784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075652.072771:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075652.072775:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075652.072777:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075652.072782:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2637 00000100:00080000:3.0:1769075652.072786:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075652.072790:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075652.072794:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075652.072804:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616422912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075652.072811:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075652.072815:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1769075652.072816:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075652.072817:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075652.072822:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2637 00000100:00080000:3.0:1769075652.072826:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075652.072829:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075652.072832:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075652.072840:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616423040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075652.072851:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075652.072855:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075652/real 1769075652] req@ffff995b84559880 x1855007616422656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075652.072856:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075652.072859:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075652.072867:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2637 00000100:00080000:3.0:1769075652.072872:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075652.072876:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075652.072880:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075652.072883:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616422656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075652.072890:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616423168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075652.072894:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075652.072903:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075652.072904:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075652.072924:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075652.072926:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075652.072929:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2637 00000100:00100000:0.0:1769075652.072932:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616422656:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075652.072933:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075652.072940:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075652.072944:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075652.072953:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616423296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075652.072960:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422784:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075652.072963:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075652.072967:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075652.072968:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075652.072972:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2637 00000100:00080000:3.0:1769075652.072976:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075652.072979:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075652.072983:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769075652.072984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075652.072994:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616423424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075652.073000:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075652/real 1769075652] req@ffff995b84558700 x1855007616422784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075652.073012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616422784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075652.073019:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075652.073022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075652.073024:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075652.073029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075652.073040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075652.073046:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075652/real 1769075652] req@ffff995b8455a300 x1855007616422912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075652.073053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616422912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075652.073060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075652.073062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075652.073064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616422912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075652.073067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616423040:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075652.073078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075652.073084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075652/real 1769075652] req@ffff995b8455b100 x1855007616423040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075652.073091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616423040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075652.073101:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075652.073103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075652.073104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616423040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075652.073107:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616423168:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075652.073116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075652.073121:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075652/real 1769075652] req@ffff995b8455ad80 x1855007616423168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075652.073128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616423168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075652.073133:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075652.073135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075652.073137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616423168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075652.073151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616423296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075652.073160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075652.073166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075652/real 1769075652] req@ffff995b84559c00 x1855007616423296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075652.073173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616423296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075652.073178:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075652.073180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075652.073182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616423296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075652.073186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616423424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075652.073196:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075652.073201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075652/real 1769075652] req@ffff995b84559880 x1855007616423424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075652.073208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616423424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075652.073213:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075652.073215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075652.073217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616423424:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769075652.776698:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075653.096538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075653.096557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075653.096563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075653.800643:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075654.120587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075654.120588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075654.120640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769075654.824616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075655.144550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075655.144551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075655.848626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075656.168518:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075656.168529:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075656.168595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075656.872677:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075657.192543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769075657.192575:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075657.192583:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075657.192590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075657.192595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075657.192601:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2642 00000100:00100000:3.0:1769075657.192604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075657.192618:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075657.192626:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075657.192668:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616423552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075657.192699:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075657.192703:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075657.192705:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075657.192710:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2642 00000100:00080000:0.0:1769075657.192714:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075657.192719:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075657.192729:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075657.192744:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616423680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075657.192761:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075657.192765:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075657.192767:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075657.192771:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2642 00000100:00080000:0.0:1769075657.192775:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075657.192779:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075657.192784:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075657.192796:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616423808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075657.192805:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075657.192809:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075657.192811:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075657.192817:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2642 00000100:00080000:0.0:1769075657.192836:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075657.192843:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075657.192849:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075657.192862:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616423936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075657.192873:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075657.192877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075657.192881:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075657.192889:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2642 00000100:00080000:0.0:1769075657.192897:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075657.192904:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075657.192910:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075657.192924:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616424064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075657.192947:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075657.192955:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075657.192958:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075657.192963:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2642 00000100:00080000:0.0:1769075657.192968:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075657.192975:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075657.192986:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075657.193001:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616424192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075657.193014:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075657.193019:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075657.193022:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075657.193028:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2642 00000100:00080000:0.0:1769075657.193035:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075657.193046:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075657.193054:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075657.193070:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616424320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075657.896578:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075658.216599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075658.216633:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075658.216670:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769075658.920631:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075659.240539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075659.240624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075659.944630:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075660.264518:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075660.264536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075660.264560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075660.264577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075660.968607:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075661.288537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075661.288560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075661.288576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769075661.992634:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075662.312544:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769075662.312556:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075662.312557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075662.312565:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075662.312570:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075662.312579:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2648 00000100:00080000:0.0:1769075662.312590:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075662.312599:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075662.312641:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616424448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075662.312664:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075662.312667:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075662.312668:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075662.312671:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2648 00000100:00080000:0.0:1769075662.312675:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075662.312678:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075662.312686:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075662.312700:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616424576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075662.312709:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075662.312712:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075662.312713:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075662.312716:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2648 00000100:00080000:0.0:1769075662.312727:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075662.312730:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075662.312734:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075662.312757:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616424704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075662.312765:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075662.312767:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075662.312768:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075662.312773:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2648 00000100:00080000:0.0:1769075662.312780:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075662.312788:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075662.312795:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075662.312810:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616424832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075662.312835:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075662.312838:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075662.312842:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075662.312850:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2648 00000100:00080000:0.0:1769075662.312857:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075662.312863:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075662.312870:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075662.312886:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616424960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075662.312894:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075662.312897:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075662.312898:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075662.312903:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2648 00000100:00080000:0.0:1769075662.312909:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075662.312912:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075662.312915:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075662.312929:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616425088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075662.312937:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075662.312941:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075662.312943:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075662.312952:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2648 00000100:00080000:0.0:1769075662.312958:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075662.312964:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075662.312971:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075662.312985:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616425216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075663.016632:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075663.336631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075663.336643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075664.040647:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075664.360553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075664.360553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075664.360603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075665.064613:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075665.384581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075665.384581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075665.384639:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075666.088651:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075666.408582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075666.408582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075666.408628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075667.112607:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075667.432572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075667.432574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075667.432611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075667.432756:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616425344:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075667.432871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616425472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075667.432914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616425600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075667.432941:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616425728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075667.432962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616425856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075667.433011:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616425984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075667.433043:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616426112:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769075668.136632:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075668.456562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075668.456649:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075668.456652:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769075669.160612:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075669.480564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075669.480573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075669.480638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769075670.184601:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075670.504561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075670.504737:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769075670.504820:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769075670.504885:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769075670.504909:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075670.504962:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075670.504974:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075670.504984:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075670.504998:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075670.505013:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075670.505035:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00010000:00010000:1.0:1769075671.208606:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075671.528546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075671.528665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075671.528712:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769075672.232611:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075672.552555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075672.552557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075672.552599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075672.552645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075672.552694:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616426240:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769075672.552755:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075672.552782:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2658 00000100:00080000:2.0:1769075672.552786:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075672.552789:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075672/real 1769075672] req@ffff995b85ac8380 x1855007616426240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075672.552798:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075672.552808:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075672.552814:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616426240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075672.552826:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769075672.552826:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616426624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075672.552836:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075672.552844:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616426240:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075672.552847:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075672.552853:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075672.552856:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075672.552861:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2658 00000100:00080000:2.0:1769075672.552866:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075672.552870:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075672.552876:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616426368:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769075672.552877:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075672.552898:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616426752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075672.552911:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075672.552912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075672.552918:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075672.552921:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075672.552927:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2658 00000100:00100000:1.0:1769075672.552933:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075672/real 1769075672] req@ffff995b85ac9880 x1855007616426368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075672.552934:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075672.552944:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075672.552953:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075672.552960:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616426368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075672.552971:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769075672.552972:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616426880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075672.552979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075672.552988:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616426368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075672.553001:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616426496:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075672.553008:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075672.553013:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075672.553015:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075672.553021:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2658 00000100:00080000:2.0:1769075672.553026:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075672.553031:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075672.553036:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075672.553043:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075672.553053:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616427008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075672.553059:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075672/real 1769075672] req@ffff995b85acb800 x1855007616426496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075672.553083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616426496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075672.553095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075672.553099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075672.553102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616426496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075672.553109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616426624:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075672.553149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075672.553161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075672/real 1769075672] req@ffff995b85aca300 x1855007616426624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075672.553173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616426624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075672.553183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075672.553187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075672.553197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616426624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075672.553217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616426752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075672.553239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075672.553251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075672/real 1769075672] req@ffff995b85acbb80 x1855007616426752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075672.553268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616426752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075672.553276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075672.553279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075672.553283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616426752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075672.553288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616426880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075672.553306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075672.553316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075672/real 1769075672] req@ffff995b85ac8a80 x1855007616426880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075672.553328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616426880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075672.553337:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075672.553340:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075672.553343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616426880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075672.553349:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616427008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075672.553369:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075672.553380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075672/real 1769075672] req@ffff995b85ac9c00 x1855007616427008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075672.553392:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616427008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075672.553408:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075672.553413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075672.553416:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616427008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075673.256578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075673.256639:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075673.576689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075673.576690:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075673.576750:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075673.576833:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075674.280560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075674.280689:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075674.600600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075674.600600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075674.600660:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075674.600691:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075675.304538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075675.304622:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075675.624557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075675.624663:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075675.624835:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075675.624879:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075676.328620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075676.328711:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075676.648584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075676.648618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075676.648619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075676.648670:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075677.352533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075677.352690:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075677.672596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075677.672596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075677.672597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769075677.672633:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075677.672642:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075677.672648:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075677.672657:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2663 00000100:00080000:2.0:1769075677.672670:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075677.672678:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075677.672679:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075677.672723:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616427136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075677.672749:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075677.672753:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075677.672754:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075677.672757:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2663 00000100:00080000:2.0:1769075677.672760:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075677.672763:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075677.672770:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075677.672780:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616427264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075677.672789:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616427136:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075677.672805:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075677.672809:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075677.672812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075677.672820:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2663 00000100:00080000:2.0:1769075677.672825:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075677.672831:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075677.672838:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075677.672855:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616427392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075677.672863:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075677.672866:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075677.672868:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075677.672871:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2663 00000100:00080000:2.0:1769075677.672877:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075677.672881:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075677.672885:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075677.672892:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075677.672905:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616427520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075677.672912:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075677.672914:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075677.672916:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075677.672920:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2663 00000100:00080000:2.0:1769075677.672922:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075677.672925:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075677.672929:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075677.672935:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075677/real 1769075677] req@ffff995b85acaa00 x1855007616427136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075677.672940:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616427648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075677.672989:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075677.672994:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075677.672995:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075677.673001:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2663 00000100:00080000:2.0:1769075677.673003:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075677.673007:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075677.673014:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616427136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769075677.673016:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075677.673034:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769075677.673036:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616427776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075677.673042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075677.673054:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616427136:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075677.673079:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075677.673082:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075677.673084:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075677.673093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616427264:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075677.673093:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2663 00000100:00080000:2.0:1769075677.673107:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075677.673114:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075677.673120:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075677.673134:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616427904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075677.673137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075677.673161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075677/real 1769075677] req@ffff995b85acb100 x1855007616427264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075677.673182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616427264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075677.673189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075677.673191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075677.673194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616427264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075677.673200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616427392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075677.673212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075677.673219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075677/real 1769075677] req@ffff995b85ac9c00 x1855007616427392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075677.673227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616427392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075677.673235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075677.673237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075677.673238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616427392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075677.673245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616427520:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075677.673266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075677.673277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075677/real 1769075677] req@ffff995b85ac8a80 x1855007616427520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075677.673291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616427520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075677.673304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075677.673306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075677.673309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616427520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075677.673313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616427648:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075677.673329:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075677.673337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075677/real 1769075677] req@ffff995b85acbb80 x1855007616427648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075677.673344:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616427648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075677.673351:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075677.673354:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075677.673356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616427648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075677.673360:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616427776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075677.673370:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075677.673376:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075677/real 1769075677] req@ffff995b85aca300 x1855007616427776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075677.673384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616427776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075677.673389:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075677.673391:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075677.673394:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616427776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075677.673410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616427904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075677.673421:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075677.673428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075677/real 1769075677] req@ffff995b85acb800 x1855007616427904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075677.673436:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616427904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075677.673442:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075677.673443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075677.673445:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616427904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075678.376551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075678.376617:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075678.696577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075678.696639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075678.696665:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075678.696707:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075679.400554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075679.400621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075679.720568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075679.720571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075679.720571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075679.720591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075680.424586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075680.424658:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075680.744536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075680.744536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075680.744601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075680.744623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075681.448585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075681.448655:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075681.768599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075681.768599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075681.768609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075681.768664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075682.472546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075682.472610:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075682.792535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075682.792547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075682.792579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075682.792598:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075682.792606:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075682.792611:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075682.792614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075682.792625:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2668 00000100:00080000:3.0:1769075682.792638:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075682.792649:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075682.792691:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616428032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075682.792714:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075682.792718:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075682.792720:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075682.792723:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2668 00000100:00080000:3.0:1769075682.792726:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075682.792730:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075682.792740:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075682.792751:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616428160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075682.792759:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075682.792763:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616428032:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075682.792770:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075682.792772:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075682.792776:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2668 00000100:00080000:3.0:1769075682.792778:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075682.792781:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075682.792784:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075682.792793:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616428288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075682.792800:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075682.792803:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075682.792805:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075682.792807:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2668 00000100:00080000:3.0:1769075682.792809:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075682.792812:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075682.792815:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075682.792822:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616428416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075682.792841:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075682.792844:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075682.792847:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769075682.792851:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075682.792855:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2668 00000100:00080000:3.0:1769075682.792862:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075682.792868:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075682.792875:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075682.792888:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616428544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075682.792895:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075682.792898:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075682.792900:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075682.792904:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2668 00000100:00100000:1.0:1769075682.792908:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075682/real 1769075682] req@ffff995b8455b480 x1855007616428032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075682.792909:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075682.792916:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075682.792921:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075682.792935:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616428672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075682.792942:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075682.792943:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616428032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075682.792947:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075682.792951:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075682.792958:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2668 00000100:00080000:3.0:1769075682.792964:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075682.792966:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075682.792967:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075682.792975:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075682.792979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075682.792986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616428032:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075682.792987:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616428800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075682.793026:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616428160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075682.793053:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075682.793067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075682/real 1769075682] req@ffff995b84558000 x1855007616428160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075682.793080:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616428160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075682.793089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075682.793092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075682.793096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616428160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075682.793102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616428288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075682.793119:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075682.793129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075682/real 1769075682] req@ffff995b8455a680 x1855007616428288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075682.793141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616428288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075682.793152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075682.793156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075682.793159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616428288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075682.793166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616428416:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075682.793187:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075682.793197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075682/real 1769075682] req@ffff995b84558a80 x1855007616428416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075682.793209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616428416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075682.793219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075682.793222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075682.793226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616428416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075682.793232:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616428544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075682.793250:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075682.793261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075682/real 1769075682] req@ffff995b8455bb80 x1855007616428544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075682.793271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616428544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075682.793281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075682.793284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075682.793287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616428544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075682.793293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616428672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075682.793311:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075682.793322:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075682/real 1769075682] req@ffff995b8455aa00 x1855007616428672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075682.793333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616428672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075682.793348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075682.793351:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075682.793354:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616428672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075682.793360:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616428800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075682.793378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075682.793388:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075682/real 1769075682] req@ffff995b84559500 x1855007616428800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075682.793415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616428800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075682.793423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075682.793427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075682.793430:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616428800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075683.496586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075683.496727:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075683.816505:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075683.816582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075683.816605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075683.816727:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075684.520537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075684.520609:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075684.840533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075684.840555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075684.840562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075684.840581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075685.544544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075685.544612:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075685.864572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075685.864578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075685.864591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075685.864603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075686.568575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075686.568674:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075686.888562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075686.888568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075686.888576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075686.888587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075687.592567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075687.592642:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075687.912545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075687.912546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075687.912559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075687.912584:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075687.912590:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075687.912594:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075687.912601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075687.912606:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2673 00000100:00080000:3.0:1769075687.912619:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075687.912627:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075687.912672:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616428928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075687.912698:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075687.912702:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075687.912703:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075687.912707:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2673 00000100:00080000:3.0:1769075687.912710:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075687.912713:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075687.912724:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075687.912733:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616428928:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075687.912737:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616429056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075687.912749:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075687.912752:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075687.912755:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075687.912760:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2673 00000100:00080000:3.0:1769075687.912764:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075687.912768:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075687.912775:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075687.912789:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616429184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075687.912810:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075687.912816:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075687.912816:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075687.912820:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075687.912826:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2673 00000100:00080000:3.0:1769075687.912829:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075687.912833:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075687.912838:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075687.912848:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616429312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075687.912855:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075687.912858:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075687.912859:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075687.912862:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2673 00000100:00080000:3.0:1769075687.912866:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075687.912868:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075687.912872:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075687.912876:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075687/real 1769075687] req@ffff995b84559500 x1855007616428928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075687.912883:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616429440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075687.912901:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075687.912904:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075687.912906:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075687.912908:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616428928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075687.912913:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2673 00000100:00080000:3.0:1769075687.912920:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075687.912926:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769075687.912927:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075687.912930:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075687.912934:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075687.912939:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616428928:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075687.912942:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616429568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075687.912949:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075687.912952:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075687.912953:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075687.912956:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2673 00000100:00080000:3.0:1769075687.912959:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075687.912961:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075687.912963:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075687.912969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075687.912974:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616429696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075687.912993:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075687.913006:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075687/real 1769075687] req@ffff995b8455aa00 x1855007616429056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075687.913015:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616429056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075687.913022:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075687.913025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075687.913027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075687.913032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075687.913042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075687.913048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075687/real 1769075687] req@ffff995b8455bb80 x1855007616429184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075687.913056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616429184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075687.913069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075687.913071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075687.913073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075687.913076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429312:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075687.913087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075687.913092:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075687/real 1769075687] req@ffff995b84558a80 x1855007616429312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075687.913099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616429312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075687.913106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075687.913108:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075687.913110:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075687.913113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075687.913122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075687.913128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075687/real 1769075687] req@ffff995b8455a680 x1855007616429440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075687.913135:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616429440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075687.913142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075687.913143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075687.913146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075687.913150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616429568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075687.913160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075687.913166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075687/real 1769075687] req@ffff995b84558000 x1855007616429568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075687.913173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616429568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075687.913179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075687.913180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075687.913182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616429568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075687.913195:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616429696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075687.913205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075687.913211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075687/real 1769075687] req@ffff995b8455b480 x1855007616429696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075687.913218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616429696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075687.913223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075687.913225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075687.913227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616429696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075688.616533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075688.616604:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075688.936551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075688.936557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075688.936599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075688.936639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075689.640574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075689.640613:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075689.960582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075689.960587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075689.960597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075689.960653:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075690.664578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075690.664688:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075690.984600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075690.984666:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075690.984681:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075690.984701:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075691.688577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075691.688655:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075692.008537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075692.008554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075692.008571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075692.008586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075692.712608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075692.712647:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075693.032557:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075693.032559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075693.032568:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075693.032572:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075693.032580:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2678 00000100:00100000:0.0:1769075693.032582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075693.032594:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075693.032596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075693.032602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075693.032606:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075693.032652:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616429824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075693.032670:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075693.032673:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075693.032675:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075693.032679:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2678 00000100:00080000:3.0:1769075693.032683:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075693.032686:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075693.032695:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075693.032696:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616429824:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075693.032714:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616429952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075693.032733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075693.032737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075693.032739:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075693.032743:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2678 00000100:00080000:3.0:1769075693.032748:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075693.032752:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769075693.032757:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769075693.032759:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075693.032775:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616430080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075693.032786:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075693.032789:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769075693.032792:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075693/real 1769075693] req@ffff995b84559880 x1855007616429824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075693.032794:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075693.032805:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2678 00000100:00080000:3.0:1769075693.032810:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075693.032815:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075693.032821:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075693.032826:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616429824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075693.032834:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616430208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075693.032836:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075693.032848:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075693.032849:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075693.032855:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769075693.032858:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616429824:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075693.032859:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075693.032867:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2678 00000100:00080000:3.0:1769075693.032871:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075693.032873:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075693.032876:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075693.032883:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616430336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075693.032889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429952:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075693.032893:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075693.032898:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075693.032899:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075693.032903:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2678 00000100:00080000:3.0:1769075693.032906:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075693.032908:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075693.032911:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075693.032920:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616430464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075693.032928:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075693.032931:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075693.032932:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075693.032935:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2678 00000800:00000100:1.0:1769075693.032938:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075693.032940:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075693.032945:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075693.032948:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075693.032957:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616430592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075693.032961:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075693/real 1769075693] req@ffff995b84559c00 x1855007616429952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075693.033000:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616429952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075693.033009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075693.033013:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075693.033016:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616429952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075693.033024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075693.033039:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075693.033046:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075693/real 1769075693] req@ffff995b8455ad80 x1855007616430080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075693.033053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616430080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075693.033060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075693.033062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075693.033064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075693.033068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430208:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075693.033078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075693.033084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075693/real 1769075693] req@ffff995b8455b100 x1855007616430208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075693.033099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616430208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075693.033106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075693.033108:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075693.033110:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430208:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075693.033123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430336:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075693.033137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075693.033144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075693/real 1769075693] req@ffff995b8455a300 x1855007616430336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075693.033152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616430336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075693.033158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075693.033159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075693.033161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075693.033165:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616430464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075693.033174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075693.033180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075693/real 1769075693] req@ffff995b84558700 x1855007616430464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075693.033188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616430464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075693.033193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075693.033194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075693.033196:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616430464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075693.033199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616430592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075693.033208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075693.033213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075693/real 1769075693] req@ffff995b84559880 x1855007616430592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075693.033220:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616430592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075693.033249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075693.033251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075693.033254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616430592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075693.736543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075693.736596:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075694.056561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075694.056599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075694.056600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075694.056633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075694.760579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075694.760703:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075695.080572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075695.080585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075695.080588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075695.080619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075695.784599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075695.784773:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075696.104641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075696.104644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075696.104647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075696.104739:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075696.808589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075696.808673:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075697.128589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075697.128610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075697.128621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075697.128629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075697.832569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075697.832736:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075698.152604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075698.152625:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075698.152635:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075698.152642:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075698.152658:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2683 00000100:00100000:2.0:1769075698.152685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075698.152685:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075698.152706:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075698.152709:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075698.152759:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075698.152778:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616430720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075698.152810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075698.152815:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075698.152817:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075698.152822:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2683 00000100:00080000:3.0:1769075698.152828:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075698.152837:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075698.152848:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075698.152861:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616430848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075698.152885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616430720:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075698.152887:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075698.152896:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075698.152901:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075698.152907:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2683 00000100:00080000:3.0:1769075698.152910:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075698.152913:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075698.152919:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075698.152938:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616430976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075698.152953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075698.152959:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075698.152962:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075698.152968:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2683 00000100:00080000:3.0:1769075698.152975:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075698.152985:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769075698.152989:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769075698.152996:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075698.153036:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616431104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075698.153048:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075698.153051:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075698.153054:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075698.153059:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2683 00000100:00080000:3.0:1769075698.153065:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075698.153067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075698/real 1769075698] req@ffff995b8455b480 x1855007616430720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075698.153072:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075698.153099:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075698.153111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616430720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075698.153114:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616431232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075698.153164:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075698.153168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075698.153172:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075698.153182:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2683 00000100:00080000:3.0:1769075698.153188:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075698.153201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075698.153202:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075698.153214:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075698.153221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075698.153232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616430720:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075698.153233:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616431360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075698.153249:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075698.153252:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075698.153254:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075698.153259:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2683 00000100:00080000:3.0:1769075698.153262:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075698.153265:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075698.153269:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075698.153278:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616431488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075698.153285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075698.153325:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075698.153341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075698/real 1769075698] req@ffff995b84558000 x1855007616430848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075698.153354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616430848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075698.153363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075698.153366:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075698.153370:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075698.153376:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075698.153392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075698.153401:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075698/real 1769075698] req@ffff995b8455a680 x1855007616430976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075698.153412:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616430976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075698.153421:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075698.153424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075698.153427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616430976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075698.153433:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616431104:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075698.153447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075698.153491:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075698/real 1769075698] req@ffff995b84558a80 x1855007616431104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075698.153503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616431104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075698.153512:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075698.153515:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075698.153518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616431104:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075698.153524:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616431232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075698.153542:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075698.153551:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075698/real 1769075698] req@ffff995b8455bb80 x1855007616431232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075698.153561:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616431232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075698.153570:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075698.153573:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075698.153576:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616431232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075698.153580:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616431360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075698.153594:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075698.153612:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075698/real 1769075698] req@ffff995b8455aa00 x1855007616431360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075698.153622:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616431360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075698.153629:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075698.153632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075698.153635:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616431360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075698.153654:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616431488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075698.153680:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075698.153689:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075698/real 1769075698] req@ffff995b84559500 x1855007616431488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075698.153700:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616431488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075698.153708:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075698.153711:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075698.153714:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616431488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075698.856527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075698.856595:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075699.176566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075699.176566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075699.176574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075699.176595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075699.880521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075699.880587:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075700.200531:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075700.200531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075700.200546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075700.200558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075700.904524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075700.904595:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075701.224526:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075701.224532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075701.224536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075701.224538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075701.928573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075701.928618:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075702.248548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075702.248552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075702.248569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075702.248662:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075702.952547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075702.952684:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075703.272572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075703.272572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769075703.272596:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075703.272606:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075703.272612:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075703.272623:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2688 00000100:00100000:3.0:1769075703.272630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075703.272636:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075703.272647:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075703.272706:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616431616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769075703.273948:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769075703.274009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075703.274015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075703.274017:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075703.274022:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2688 00000100:00080000:0.0:1769075703.274028:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075703.274032:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075703.274042:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075703.274072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616431616:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075703.274091:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616431744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075703.274117:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075703.274119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075703.274121:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075703.274124:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2688 00000100:00080000:0.0:1769075703.274127:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075703.274130:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075703.274134:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075703.274145:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616431872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075703.274152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075703.274154:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075703.274155:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075703.274158:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2688 00000100:00080000:0.0:1769075703.274161:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075703.274164:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769075703.274170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769075703.274171:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075703.274182:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616432000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075703.274189:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075703.274191:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075703.274193:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075703.274206:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2688 00000100:00080000:0.0:1769075703.274209:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075703.274212:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075703.274216:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075703.274219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075703/real 1769075703] req@ffff995b83505180 x1855007616431616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075703.274227:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616432128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075703.274236:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075703.274239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1769075703.274241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616431616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075703.274242:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075703.274251:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2688 00000100:00080000:0.0:1769075703.274257:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075703.274260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075703.274261:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075703.274269:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769075703.274272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075703.274277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616431616:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075703.274280:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616432256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075703.274292:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075703.274293:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075703.274295:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075703.274298:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2688 00000100:00080000:0.0:1769075703.274301:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075703.274302:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616431744:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769075703.274308:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075703.274313:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075703.274322:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616432384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075703.274330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075703.274345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075703/real 1769075703] req@ffff995b83504e00 x1855007616431744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075703.274357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616431744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075703.274363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075703.274366:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075703.274368:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616431744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075703.274374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616431872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075703.274388:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075703.274396:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075703/real 1769075703] req@ffff995b83507800 x1855007616431872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075703.274406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616431872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075703.274418:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075703.274421:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075703.274424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616431872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075703.274428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432000:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075703.274443:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075703.274454:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075703/real 1769075703] req@ffff995b83506300 x1855007616432000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075703.274479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616432000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075703.274511:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075703.274516:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075703.274520:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075703.274528:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432128:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075703.274546:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075703.274556:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075703/real 1769075703] req@ffff995b83507100 x1855007616432128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075703.274564:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616432128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075703.274570:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075703.274572:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075703.274575:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075703.274579:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616432256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075703.274589:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075703.274596:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075703/real 1769075703] req@ffff995b83504000 x1855007616432256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075703.274608:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616432256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075703.274616:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075703.274620:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075703.274623:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616432256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075703.274643:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616432384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075703.274666:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075703.274676:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075703/real 1769075703] req@ffff995b83505180 x1855007616432384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075703.274684:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616432384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075703.274690:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075703.274693:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075703.274695:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616432384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075703.976646:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075703.976700:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075704.296627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075704.296628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075704.296686:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075704.296715:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075705.000545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075705.000611:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075705.320557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075705.320559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075705.320592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075705.320593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075706.024607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075706.024655:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075706.344586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075706.344588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075706.344614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075706.344640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075707.048635:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075707.048791:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075707.368588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075707.368596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075707.368611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075707.368682:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075708.072544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075708.072704:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075708.392554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075708.392554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075708.392577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075708.392589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075708.392611:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075708.392618:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075708.392623:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075708.392632:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2694 00000100:00080000:3.0:1769075708.392644:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075708.392693:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075708.392726:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616432512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075708.392768:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075708.392773:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075708.392776:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075708.392782:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2694 00000100:00080000:3.0:1769075708.392788:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075708.392794:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075708.392824:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075708.392835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616432512:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075708.392844:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616432640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075708.392867:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075708.392871:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075708.392873:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075708.392877:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2694 00000100:00080000:3.0:1769075708.392881:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075708.392884:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075708.392890:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075708.392903:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616432768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075708.392913:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075708.392917:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075708.392919:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075708.392923:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2694 00000100:00080000:3.0:1769075708.392943:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075708.392947:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769075708.392954:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769075708.392956:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075708.393010:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616432896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075708.393034:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075708/real 1769075708] req@ffff995b84559500 x1855007616432512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075708.393050:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075708.393054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075708.393055:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075708.393060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616432512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075708.393062:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2694 00000100:00080000:3.0:1769075708.393069:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075708.393073:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075708.393075:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769075708.393082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769075708.393084:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075708.393091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616432512:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075708.393102:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616433024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075708.393114:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075708.393118:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075708.393121:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075708.393122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432640:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075708.393129:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2694 00000100:00080000:3.0:1769075708.393135:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075708.393140:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075708.393146:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075708.393159:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616433152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075708.393162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075708.393174:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075708.393177:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075708.393179:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075708.393183:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2694 00000100:00100000:2.0:1769075708.393186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075708/real 1769075708] req@ffff995b8455aa00 x1855007616432640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075708.393188:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075708.393193:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075708.393198:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075708.393209:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616433280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075708.393217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616432640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075708.393233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075708.393236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075708.393241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075708.393246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075708.393273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075708.393282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075708/real 1769075708] req@ffff995b8455bb80 x1855007616432768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075708.393290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616432768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075708.393297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075708.393299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075708.393302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075708.393307:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075708.393318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075708.393324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075708/real 1769075708] req@ffff995b84558a80 x1855007616432896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075708.393332:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616432896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075708.393337:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075708.393339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075708.393341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616432896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075708.393362:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433024:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075708.393379:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075708.393389:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075708/real 1769075708] req@ffff995b8455a680 x1855007616433024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075708.393400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616433024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075708.393425:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075708.393427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075708.393429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075708.393447:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616433152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075708.393463:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075708.393472:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075708/real 1769075708] req@ffff995b84558000 x1855007616433152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075708.393504:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616433152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075708.393512:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075708.393515:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075708.393518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616433152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075708.393523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616433280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075708.393539:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075708.393551:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075708/real 1769075708] req@ffff995b8455b480 x1855007616433280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075708.393580:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616433280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075708.393588:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075708.393592:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075708.393595:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616433280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075709.096569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075709.096646:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075709.416513:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075709.416517:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075709.416541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075709.416554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075710.120572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075710.120722:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075710.440538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075710.440538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075710.440572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075710.440633:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075711.144533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075711.144600:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075711.464512:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075711.464545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075711.464545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075711.464556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075712.168566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075712.168671:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075712.488618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075712.488619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075712.488723:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075712.488758:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075713.192538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075713.192651:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075713.512550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075713.512550:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075713.512553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075713.512562:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075713.512568:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075713.512577:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2699 00000100:00100000:2.0:1769075713.512579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075713.512587:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075713.512595:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075713.512601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075713.512627:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616433408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075713.512653:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075713.512665:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075713.512668:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075713.512673:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2699 00000100:00080000:3.0:1769075713.512678:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075713.512684:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616433408:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075713.512685:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075713.512700:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075713.512715:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616433536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075713.512724:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075713.512727:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075713.512728:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075713.512731:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2699 00000100:00080000:3.0:1769075713.512741:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075713.512745:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075713.512748:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075713.512752:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075713.512760:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616433664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075713.512780:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075713.512783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075713.512784:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075713.512786:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2699 00000100:00080000:3.0:1769075713.512789:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075713.512792:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075713.512796:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075713.512799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075713/real 1769075713] req@ffff995b84559880 x1855007616433408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075713.512807:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616433792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075713.512820:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616433408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075713.512821:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075713.512831:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075713.512834:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075713.512838:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075713.512839:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2699 00000100:00080000:3.0:1769075713.512845:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075713.512846:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075713.512850:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075713.512855:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616433408:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769075713.512857:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075713.512868:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616433920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075713.512874:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075713.512889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075713.512891:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075713.512897:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2699 00000100:00100000:2.0:1769075713.512898:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433536:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075713.512900:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075713.512907:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075713.512912:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075713.512923:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616434048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075713.512931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075713.512935:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075713.512944:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075713.512947:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075713.512951:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2699 00000100:00080000:3.0:1769075713.512955:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075713.512958:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075713.512963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075713/real 1769075713] req@ffff995b84558700 x1855007616433536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075713.512963:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075713.512977:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616434176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075713.512987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616433536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075713.512999:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075713.513003:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075713.513007:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075713.513013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075713.513029:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075713.513037:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075713/real 1769075713] req@ffff995b8455a300 x1855007616433664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075713.513044:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616433664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075713.513051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075713.513053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075713.513055:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075713.513059:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075713.513069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075713.513074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075713/real 1769075713] req@ffff995b8455b100 x1855007616433792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075713.513087:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616433792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075713.513093:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075713.513094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075713.513096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075713.513099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075713.513108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075713.513113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075713/real 1769075713] req@ffff995b8455ad80 x1855007616433920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075713.513120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616433920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075713.513126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075713.513127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075713.513129:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616433920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075713.513143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616434048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075713.513152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075713.513158:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075713/real 1769075713] req@ffff995b84559c00 x1855007616434048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075713.513165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616434048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075713.513174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075713.513176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075713.513178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616434048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075713.513182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616434176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075713.513191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075713.513196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075713/real 1769075713] req@ffff995b84559880 x1855007616434176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075713.513203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616434176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075713.513208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075713.513210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075713.513212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616434176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075714.216525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075714.216636:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075714.536564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075714.536579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075714.536584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075714.536640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075715.240560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075715.240620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075715.560547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075715.560556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075715.560562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075715.560582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075716.264525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075716.264626:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075716.584516:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075716.584523:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075716.584545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075716.584550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075717.288510:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075717.288636:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075717.608524:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075717.608527:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075717.608544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075717.608552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075718.312514:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075718.312571:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075718.632546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075718.632552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075718.632596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075718.632619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769075718.632647:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075718.632655:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075718.632660:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075718.632669:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2704 00000100:00080000:2.0:1769075718.632681:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075718.632688:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075718.632726:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616434304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075718.632763:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075718.632769:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075718.632781:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075718.632790:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2704 00000100:00080000:2.0:1769075718.632814:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075718.632821:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075718.632851:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075718.632872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616434304:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075718.632873:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616434432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075718.632893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075718.632897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075718.632899:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075718.632903:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2704 00000100:00080000:2.0:1769075718.632908:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075718.632911:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075718.632916:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075718.632930:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616434560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075718.632940:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075718.632944:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075718.632946:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075718.632949:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2704 00000100:00080000:2.0:1769075718.632952:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075718.632955:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075718.632961:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075718.632967:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075718.632979:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616434688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075718.632988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075718.632991:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075718.632993:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075718.632997:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2704 00000100:00080000:2.0:1769075718.633001:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075718.633009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075718/real 1769075718] req@ffff995b85ac9880 x1855007616434304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075718.633009:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075718.633016:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075718.633038:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616434304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075718.633044:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616434816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075718.633062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075718.633064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075718.633070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769075718.633072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075718.633077:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075718.633079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616434304:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075718.633084:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2704 00000100:00080000:2.0:1769075718.633089:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075718.633093:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075718.633096:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075718.633106:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616434944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075718.633115:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075718.633118:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075718.633120:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075718.633123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616434432:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075718.633128:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2704 00000100:00080000:2.0:1769075718.633133:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075718.633136:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075718.633141:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075718.633154:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616435072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075718.633161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075718.633201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075718/real 1769075718] req@ffff995b85ac8380 x1855007616434432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075718.633233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616434432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075718.633246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075718.633251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075718.633257:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616434432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075718.633267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616434560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075718.633311:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075718.633341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075718/real 1769075718] req@ffff995b85acb800 x1855007616434560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075718.633353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616434560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075718.633363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075718.633366:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075718.633369:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616434560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075718.633374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616434688:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075718.633391:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075718.633399:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075718/real 1769075718] req@ffff995b85aca300 x1855007616434688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075718.633409:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616434688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075718.633422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075718.633425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075718.633428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616434688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075718.633432:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616434816:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075718.633445:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075718.633452:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075718/real 1769075718] req@ffff995b85acbb80 x1855007616434816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075718.633463:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616434816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075718.633507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075718.633512:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075718.633518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616434816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075718.633527:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616434944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075718.633575:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075718.633588:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075718/real 1769075718] req@ffff995b85ac8a80 x1855007616434944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075718.633600:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616434944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075718.633608:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075718.633611:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075718.633615:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616434944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075718.633636:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616435072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075718.633653:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075718.633662:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075718/real 1769075718] req@ffff995b85ac9c00 x1855007616435072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075718.633673:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616435072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075718.633681:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075718.633683:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075718.633686:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616435072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075719.336528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075719.336571:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075719.656570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075719.656570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075719.656596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075719.656638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075720.360516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075720.360624:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075720.680547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075720.680593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075720.680603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075720.680627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075721.384549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075721.384597:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075721.704672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075721.704685:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075721.704719:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075721.704830:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075722.408508:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075722.408822:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075722.728646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075722.728657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075722.728701:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075722.728705:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075723.432571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075723.432731:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075723.752641:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075723.752650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075723.752650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075723.752653:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769075723.752656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075723.752662:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075723.752676:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2709 00000100:00080000:3.0:1769075723.752690:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075723.752698:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075723.752782:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075723.752800:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616435200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075723.752838:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075723.752844:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075723.752846:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075723.752851:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2709 00000100:00080000:3.0:1769075723.752857:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075723.752862:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075723.752874:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075723.752892:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616435328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075723.752905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616435200:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075723.752914:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075723.752921:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075723.752923:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075723.752929:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2709 00000100:00080000:3.0:1769075723.752934:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075723.752939:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075723.752947:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075723.752965:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616435456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075723.752978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075723.752982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075723.752986:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075723.752991:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2709 00000100:00080000:3.0:1769075723.752996:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075723.753000:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769075723.753011:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769075723.753014:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075723.753033:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616435584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075723.753046:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075723.753051:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075723.753053:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075723.753058:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2709 00000100:00080000:3.0:1769075723.753063:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075723.753068:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075723.753075:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075723.753090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075723/real 1769075723] req@ffff995b8455b480 x1855007616435200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075723.753092:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616435712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075723.753146:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075723.753158:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075723.753165:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075723.753181:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2709 00000100:00080000:3.0:1769075723.753203:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075723.753204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616435200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075723.753220:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769075723.753247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075723.753253:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075723.753257:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075723.753268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616435200:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075723.753287:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616435840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075723.753307:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075723.753313:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769075723.753317:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616435328:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075723.753320:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075723.753333:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2709 00000100:00080000:3.0:1769075723.753343:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075723.753351:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769075723.753360:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769075723.753368:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075723.753381:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075723/real 1769075723] req@ffff995b84558000 x1855007616435328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075723.753398:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616435968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075723.753421:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616435328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075723.753504:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075723.753511:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075723.753518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616435328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075723.753535:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616435456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075723.753565:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075723.753598:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075723/real 1769075723] req@ffff995b8455a680 x1855007616435456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075723.753612:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616435456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075723.753629:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075723.753633:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075723.753637:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616435456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075723.753643:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616435584:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075723.753661:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075723.753671:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075723/real 1769075723] req@ffff995b84558a80 x1855007616435584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075723.753682:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616435584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075723.753692:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075723.753695:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075723.753698:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616435584:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075723.753704:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616435712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075723.753784:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075723.753800:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075723/real 1769075723] req@ffff995b8455bb80 x1855007616435712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075723.753815:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616435712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075723.753827:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075723.753831:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075723.753836:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616435712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075723.753843:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616435840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075723.753864:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075723.753875:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075723/real 1769075723] req@ffff995b8455aa00 x1855007616435840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075723.753889:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616435840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075723.753908:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075723.753911:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075723.753915:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616435840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075723.753943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616435968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075723.753962:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075723.753974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075723/real 1769075723] req@ffff995b84559500 x1855007616435968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075723.753995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616435968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075723.754005:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075723.754008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075723.754012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616435968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075724.456664:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075724.456687:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075724.776621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075724.776631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075724.776674:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075724.776760:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075725.480603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075725.480730:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075725.800596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075725.800596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075725.800606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075725.800606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075726.504564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075726.504680:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075726.824556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075726.824572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075726.824609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075726.824609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075727.528525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075727.528586:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075727.848537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075727.848546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075727.848554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075727.848590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075728.552588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075728.552736:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075728.872516:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075728.872528:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075728.872533:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075728.872543:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2714 00000100:00100000:2.0:1769075728.872553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075728.872556:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075728.872566:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075728.872611:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616436096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075728.872615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075728.872649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075728.872652:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075728.872678:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075728.872686:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075728.872689:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075728.872697:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2714 00000100:00100000:1.0:1769075728.872698:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616436096:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075728.872704:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075728.872711:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075728.872721:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075728.872738:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616436224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075728.872750:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075728.872753:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075728.872755:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075728.872758:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2714 00000100:00080000:3.0:1769075728.872762:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075728.872765:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075728.872769:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075728.872778:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075728.872782:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616436352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075728.872792:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075728.872795:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075728.872797:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075728.872802:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2714 00000100:00080000:3.0:1769075728.872821:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075728.872824:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075728.872829:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075728.872842:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616436480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075728.872849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075728/real 1769075728] req@ffff995b84559500 x1855007616436096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075728.872855:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075728.872866:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075728.872870:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075728.872874:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2714 00000100:00080000:3.0:1769075728.872876:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075728.872879:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075728.872883:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075728.872897:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616436608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075728.872906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075728.872909:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616436096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075728.872910:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075728.872915:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075728.872925:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2714 00000100:00080000:3.0:1769075728.872932:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075728.872938:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769075728.872941:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075728.872943:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075728.872969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769075728.872975:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616436736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075728.872981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616436096:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075728.872991:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075728.872995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075728.872997:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075728.873001:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2714 00000100:00080000:3.0:1769075728.873005:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075728.873009:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075728.873012:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075728.873036:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616436864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075728.873051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616436224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075728.873093:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075728.873128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075728/real 1769075728] req@ffff995b8455aa00 x1855007616436224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075728.873147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616436224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075728.873169:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075728.873174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075728.873180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616436224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075728.873189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616436352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075728.873233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075728.873250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075728/real 1769075728] req@ffff995b8455bb80 x1855007616436352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075728.873267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616436352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075728.873281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075728.873286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075728.873291:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616436352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075728.873299:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616436480:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075728.873357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075728.873371:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075728/real 1769075728] req@ffff995b84558a80 x1855007616436480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075728.873387:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616436480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075728.873399:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075728.873404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075728.873408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616436480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075728.873415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616436608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075728.873438:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075728.873452:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075728/real 1769075728] req@ffff995b8455a680 x1855007616436608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075728.873494:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616436608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075728.873507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075728.873512:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075728.873516:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616436608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075728.873524:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616436736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075728.873548:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075728.873561:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075728/real 1769075728] req@ffff995b84558000 x1855007616436736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075728.873576:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616436736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075728.873586:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075728.873590:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075728.873594:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616436736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075728.873643:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616436864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075728.873673:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075728.873688:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075728/real 1769075728] req@ffff995b8455b480 x1855007616436864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075728.873704:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616436864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075728.873716:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075728.873720:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075728.873724:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616436864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075729.576669:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075729.576741:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075729.896635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075729.896635:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075729.896643:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075729.896694:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075730.600587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075730.600706:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075730.920607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075730.920745:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075730.920752:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075730.920823:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075731.624734:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075731.624755:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075731.944566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075731.944601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075731.944669:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075731.944709:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075732.648535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075732.648634:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075732.968587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075732.968616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075732.968634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075732.968636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075733.672557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075733.672654:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075733.992573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075733.992593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769075733.992594:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075733.992605:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075733.992611:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075733.992620:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2719 00000100:00080000:2.0:1769075733.992631:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075733.992638:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075733.992645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075733.992664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075733.992679:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616436992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075733.992716:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075733.992722:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075733.992726:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075733.992735:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2719 00000100:00080000:2.0:1769075733.992741:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075733.992745:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075733.992750:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616436992:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769075733.992756:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075733.992773:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616437120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075733.992783:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075733.992785:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075733.992787:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075733.992791:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2719 00000100:00080000:2.0:1769075733.992794:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075733.992796:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075733.992799:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075733.992826:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616437248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075733.992833:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075733.992835:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075733.992837:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075733.992840:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2719 00000100:00080000:2.0:1769075733.992843:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075733.992846:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075733.992850:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075733.992863:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616437376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075733.992894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616437120:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075733.992896:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075733.992902:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075733.992907:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075733.992912:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2719 00000100:00080000:2.0:1769075733.992916:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075733.992919:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075733.992924:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075733.992938:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616437504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075733.992946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075733.992949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075733.992951:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075733.992956:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2719 00000100:00100000:1.0:1769075733.992957:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616437248:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075733.992960:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075733.992967:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075733.992971:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075733.992983:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616437632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075733.992989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075733.992992:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075733.992993:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075733.992997:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2719 00000100:00080000:2.0:1769075733.993001:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075733.993004:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075733.993009:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075733.993014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616437376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075733.993021:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616437760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075733.993042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616437504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075733.993071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616437632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075733.993087:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616437760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075734.696541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075734.696596:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075735.016653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075735.016670:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075735.016705:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075735.016757:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075735.720654:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075735.720806:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075736.040621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075736.040677:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075736.040685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075736.040689:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075736.744591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075736.744719:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075737.064642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075737.064676:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075737.064677:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075737.064744:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:2.0:1769075737.064967:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769075737.065036:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769075737.065065:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769075737.065135:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769075737.065151:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769075737.065169:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769075737.065187:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769075737.065210:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:1.0:1769075737.065232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075733/real 1769075737] req@ffff995b85ac8a80 x1855007616437376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1769075737.065233:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:1.0:1769075737.065286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616437376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075737.065317:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075737.065327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075737.065336:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616437376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075737.065352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075733/real 1769075737] req@ffff995b85acbb80 x1855007616437504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075737.065388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616437504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075737.065406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075737.065414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075737.065421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616437504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075737.065445:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075733/real 1769075737] req@ffff995b85aca300 x1855007616437632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075737.065545:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616437632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075737.065563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075737.065571:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075737.065578:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616437632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075737.065592:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075733/real 1769075737] req@ffff995b85acaa00 x1855007616437120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075737.065631:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616437120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075737.065651:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075737.065666:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075737.065675:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616437120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075737.065688:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075733/real 1769075737] req@ffff995b85ac9c00 x1855007616437248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075737.065714:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616437248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075737.065734:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075737.065742:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075737.065749:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616437248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075737.065761:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075733/real 1769075737] req@ffff995b85acb100 x1855007616436992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075737.065799:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616436992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075737.065814:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075737.065822:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075737.065831:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616436992:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075737.065890:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075733/real 1769075737] req@ffff995b85acb800 x1855007616437760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075737.065921:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616437760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075737.065937:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075737.065944:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075737.065951:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616437760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075737.768599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075737.768698:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075738.088632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075738.088639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075738.088710:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075738.088810:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075738.792620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075738.792720:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075739.112695:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075739.112708:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075739.112749:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075739.112769:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075739.112777:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075739.112797:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2724 00000100:00100000:0.0:1769075739.112814:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769075739.112818:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075739.112837:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075739.112851:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075739.112903:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616437888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075739.112964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075739.112973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075739.112981:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075739.112998:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2724 00000100:00080000:2.0:1769075739.113012:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075739.113022:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075739.113040:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075739.113046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616437888:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075739.113079:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616438016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075739.113106:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075739.113113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075739.113117:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075739.113126:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2724 00000100:00080000:2.0:1769075739.113137:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075739.113146:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075739.113158:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075739.113182:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616438144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075739.113203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075739.113233:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075739.113241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075739.113245:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075739.113257:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2724 00000100:00080000:2.0:1769075739.113266:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075739.113276:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075739.113302:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075739.113323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075739/real 1769075739] req@ffff995b85ac8380 x1855007616437888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075739.113334:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616438272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075739.113412:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075739.113419:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075739.113423:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075739.113535:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616437888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075739.113547:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2724 00000100:00080000:2.0:1769075739.113570:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075739.113579:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075739.113589:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075739.113605:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769075739.113610:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075739.113626:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616437888:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075739.113643:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616438400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075739.113665:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075739.113671:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075739.113675:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075739.113685:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2724 00000100:00100000:1.0:1769075739.113695:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438016:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075739.113701:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075739.113721:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075739.113732:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075739.113758:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616438528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075739.113776:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075739.113791:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075739.113795:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075739.113804:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075739.113817:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2724 00000100:00080000:2.0:1769075739.113830:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075739.113846:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075739/real 1769075739] req@ffff995b85ac9880 x1855007616438016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075739.113846:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075739.113866:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075739.113914:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616438656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075739.113916:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616438016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075739.113996:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075739.114010:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075739.114019:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075739.114043:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075739.114114:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075739.114145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075739/real 1769075739] req@ffff995b85acb800 x1855007616438144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075739.114175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616438144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075739.114208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075739.114217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075739.114224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075739.114238:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075739.114293:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075739.114329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075739/real 1769075739] req@ffff995b85acb100 x1855007616438272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075739.114382:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616438272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075739.114404:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075739.114411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075739.114418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075739.114434:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075739.114578:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075739.114605:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075739/real 1769075739] req@ffff995b85ac9c00 x1855007616438400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075739.114636:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616438400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075739.114656:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075739.114666:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075739.114676:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075739.114738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616438528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075739.114801:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075739.114829:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075739/real 1769075739] req@ffff995b85acaa00 x1855007616438528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075739.114865:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616438528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075739.114914:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075739.114942:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075739.114954:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616438528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075739.114978:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616438656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075739.115037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075739.115086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075739/real 1769075739] req@ffff995b85aca300 x1855007616438656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075739.115120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616438656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075739.115139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075739.115152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075739.115160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616438656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075739.816566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075739.816784:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075740.136542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075740.136546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075740.136566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075740.136625:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075740.840579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075740.840622:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075741.160577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075741.160580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075741.160598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075741.160736:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075741.864542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075741.864723:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075742.184581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075742.184603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075742.184606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075742.184648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075742.888559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075742.888663:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075743.208626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075743.208657:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075743.208666:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075743.208707:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075743.912560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075743.912680:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075744.232612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075744.232620:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075744.232622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075744.232632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075744.232637:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075744.232648:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075744.232666:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2729 00000100:00100000:0.0:1769075744.232669:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075744.232689:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075744.232705:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075744.232760:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616438784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075744.232807:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075744.232814:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075744.232819:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075744.232827:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2729 00000100:00080000:3.0:1769075744.232835:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075744.232844:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075744.232859:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075744.232875:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616438784:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075744.232884:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616438912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075744.232904:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075744.232910:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075744.232913:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075744.232919:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2729 00000100:00080000:3.0:1769075744.232925:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075744.232933:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075744.232941:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075744.232956:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616439040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075744.232969:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075744.232973:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075744.232979:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075744.232982:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075744.232989:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2729 00000100:00080000:3.0:1769075744.232994:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075744.233001:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075744.233008:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075744.233027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075744/real 1769075744] req@ffff995b84559880 x1855007616438784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075744.233027:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616439168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075744.233047:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075744.233053:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075744.233056:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075744.233061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616438784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075744.233063:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2729 00000100:00080000:3.0:1769075744.233069:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075744.233076:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769075744.233080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075744.233084:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769075744.233088:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075744.233095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616438784:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075744.233100:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616439296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075744.233115:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075744.233121:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075744.233124:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075744.233130:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2729 00000100:00100000:1.0:1769075744.233135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438912:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075744.233137:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075744.233144:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075744.233152:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075744.233176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075744.233191:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616439424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075744.233201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075744/real 1769075744] req@ffff995b84559c00 x1855007616438912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075744.233212:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075744.233219:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075744.233223:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075744.233226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616438912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075744.233230:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2729 00000100:00080000:3.0:1769075744.233237:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075744.233244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075744.233244:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769075744.233251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769075744.233254:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075744.233257:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616438912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075744.233269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075744.233273:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616439552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075744.233303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075744.233326:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075744/real 1769075744] req@ffff995b8455ad80 x1855007616439040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075744.233351:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616439040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075744.233420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075744.233424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075744.233429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075744.233438:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439168:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075744.233502:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075744.233518:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075744/real 1769075744] req@ffff995b8455b100 x1855007616439168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075744.233533:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616439168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075744.233545:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075744.233549:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075744.233553:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075744.233560:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439296:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075744.233581:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075744.233592:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075744/real 1769075744] req@ffff995b8455a300 x1855007616439296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075744.233606:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616439296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075744.233617:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075744.233621:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075744.233626:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075744.233652:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616439424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075744.233672:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075744.233684:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075744/real 1769075744] req@ffff995b84558700 x1855007616439424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075744.233698:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616439424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075744.233708:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075744.233711:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075744.233715:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616439424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075744.233722:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616439552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075744.233739:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075744.233750:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075744/real 1769075744] req@ffff995b84559880 x1855007616439552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075744.233764:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616439552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075744.233773:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075744.233777:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075744.233780:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616439552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075744.936613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075744.936739:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075745.256546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075745.256586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075745.256599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075745.256708:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075745.960592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075745.960711:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075746.280701:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075746.280787:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075746.280801:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075746.280929:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075746.984549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075746.984707:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075747.304587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075747.304605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075747.304625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075747.304665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075748.008587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075748.008708:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075748.328562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075748.328585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075748.328613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075748.328662:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075749.032588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075749.032695:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075749.352572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075749.352582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075749.352622:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075749.352631:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075749.352639:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075749.352652:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2735 00000100:00100000:1.0:1769075749.352659:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075749.352662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075749.352664:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075749.352678:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075749.352713:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616439680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075749.352741:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075749.352745:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075749.352748:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075749.352756:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2735 00000100:00080000:3.0:1769075749.352762:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075749.352764:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616439680:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075749.352773:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075749.352786:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075749.352806:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616439808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075749.352838:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075749.352842:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1769075749.352844:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075749.352845:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075749.352853:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2735 00000100:00080000:3.0:1769075749.352858:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075749.352862:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075749.352867:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075749.352879:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616439936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075749.352887:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075749/real 1769075749] req@ffff995b8455b480 x1855007616439680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075749.352911:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075749.352916:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075749.352918:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075749.352922:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2735 00000100:00080000:3.0:1769075749.352925:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075749.352931:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075749.352933:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616439680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075749.352936:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075749.352950:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616440064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075749.352956:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075749.352961:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075749.352964:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075749.352969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075749.352969:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075749.352978:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2735 00000100:00080000:3.0:1769075749.352982:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075749.352983:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616439680:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075749.352987:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075749.352994:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075749.353003:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616440192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075749.353010:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075749.353013:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075749.353014:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075749.353018:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2735 00000100:00080000:3.0:1769075749.353022:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075749.353027:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439808:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075749.353029:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075749.353035:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075749.353050:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616440320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075749.353057:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075749.353060:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075749.353061:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075749.353064:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2735 00000100:00080000:3.0:1769075749.353067:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075749.353070:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075749.353074:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075749.353085:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075749.353086:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616440448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075749.353104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075749/real 1769075749] req@ffff995b84558000 x1855007616439808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075749.353117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616439808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075749.353125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075749.353127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075749.353131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075749.353137:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075749.353162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075749.353170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075749/real 1769075749] req@ffff995b8455a680 x1855007616439936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075749.353179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616439936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075749.353194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075749.353197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075749.353202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616439936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075749.353207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440064:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075749.353222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075749.353230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075749/real 1769075749] req@ffff995b84558a80 x1855007616440064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075749.353244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616440064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075749.353255:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075749.353259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075749.353263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440064:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075749.353270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075749.353319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075749.353335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075749/real 1769075749] req@ffff995b8455bb80 x1855007616440192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075749.353348:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616440192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075749.353356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075749.353358:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075749.353361:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075749.353380:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616440320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075749.353425:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075749.353435:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075749/real 1769075749] req@ffff995b8455aa00 x1855007616440320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075749.353445:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616440320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075749.353452:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075749.353455:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075749.353457:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616440320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075749.353462:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616440448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075749.353505:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075749.353515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075749/real 1769075749] req@ffff995b84559500 x1855007616440448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075749.353524:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616440448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075749.353532:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075749.353534:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075749.353537:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616440448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075750.056628:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075750.056713:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075750.376546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075750.376548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075750.376550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075750.376607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075751.080568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075751.080718:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075751.400529:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075751.400529:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075751.400564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075751.400566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075752.104549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075752.104644:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075752.424527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075752.424550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075752.424553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075752.424572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075753.128542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075753.128638:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075753.448546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075753.448569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075753.448586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075753.448612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075754.152564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075754.152615:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075754.472534:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075754.472534:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075754.472536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075754.472546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075754.472550:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075754.472555:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075754.472564:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2740 00000100:00080000:3.0:1769075754.472574:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075754.472579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075754.472581:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075754.472616:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616440576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075754.472636:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075754.472639:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075754.472640:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075754.472643:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2740 00000100:00080000:3.0:1769075754.472646:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075754.472649:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075754.472657:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075754.472666:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616440704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075754.472669:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616440576:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075754.472682:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075754.472684:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075754.472685:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075754.472688:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2740 00000100:00080000:3.0:1769075754.472691:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075754.472693:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075754.472697:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075754.472705:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616440832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075754.472712:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075754.472714:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075754.472715:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075754.472717:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2740 00000100:00080000:3.0:1769075754.472720:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075754.472723:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075754.472727:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769075754.472735:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075754.472740:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616440960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075754.472759:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075754.472763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075754.472765:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075754.472770:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075754/real 1769075754] req@ffff995b84559500 x1855007616440576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075754.472773:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2740 00000100:00080000:3.0:1769075754.472781:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075754.472788:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075754.472792:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616440576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075754.472796:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769075754.472805:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075754.472809:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075754.472813:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616440576:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075754.472818:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616441088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075754.472830:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075754.472834:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769075754.472835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440704:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075754.472838:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075754.472846:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2740 00000100:00080000:3.0:1769075754.472852:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075754.472861:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075754.472865:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075754.472866:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075754.472879:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616441216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075754.472882:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075754/real 1769075754] req@ffff995b8455aa00 x1855007616440704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075754.472912:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616440704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075754.472914:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075754.472923:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075754.472926:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075754.472931:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075754.472932:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2740 00000100:00080000:1.0:1769075754.472938:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075754.472939:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075754.472945:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075754.472948:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075754.472949:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075754.472958:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075754.472958:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616441344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075754.472982:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075754.472992:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075754/real 1769075754] req@ffff995b8455bb80 x1855007616440832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075754.473000:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616440832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075754.473007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075754.473009:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075754.473012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075754.473016:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440960:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075754.473026:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075754.473032:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075754/real 1769075754] req@ffff995b84558a80 x1855007616440960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075754.473039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616440960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075754.473050:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075754.473052:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075754.473054:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616440960:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075754.473057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075754.473065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075754.473071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075754/real 1769075754] req@ffff995b8455a680 x1855007616441088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075754.473078:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616441088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075754.473084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075754.473086:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075754.473087:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075754.473101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616441216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075754.473111:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075754.473127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075754/real 1769075754] req@ffff995b84558000 x1855007616441216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075754.473134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616441216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075754.473139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075754.473141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075754.473144:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616441216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075754.473147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616441344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075754.473157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075754.473162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075754/real 1769075754] req@ffff995b8455b480 x1855007616441344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075754.473169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616441344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075754.473175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075754.473176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075754.473178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616441344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075755.176522:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075755.176588:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075755.496552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075755.496552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075755.496593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075755.496604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075756.200521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075756.200565:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075756.520534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075756.520536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075756.520550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075756.520557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075757.224508:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075757.224585:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075757.544576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075757.544576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075757.544624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075757.544669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075758.248603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075758.248621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075758.568535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075758.568537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075758.568564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075758.568570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075759.272534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075759.272605:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075759.592535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075759.592536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075759.592540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075759.592564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075759.592576:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075759.592581:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075759.592585:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075759.592592:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2745 00000100:00080000:1.0:1769075759.592610:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075759.592618:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075759.592653:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616441472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075759.592680:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075759.592683:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075759.592684:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075759.592688:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2745 00000100:00080000:1.0:1769075759.592691:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075759.592695:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075759.592704:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075759.592712:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616441472:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075759.592722:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616441600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075759.592735:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075759.592739:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075759.592741:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075759.592746:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2745 00000100:00080000:1.0:1769075759.592750:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075759.592755:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075759.592760:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075759.592772:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616441728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075759.592776:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075759.592786:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075759.592789:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075759.592791:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075759.592795:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2745 00000100:00080000:1.0:1769075759.592798:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075759.592801:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075759.592805:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075759.592815:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075759/real 1769075759] req@ffff995b868c7100 x1855007616441472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075759.592817:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616441856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075759.592849:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075759.592852:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075759.592853:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075759.592857:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2745 00000100:00080000:1.0:1769075759.592860:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075759.592861:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616441472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075759.592867:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075759.592873:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769075759.592879:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769075759.592884:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616441984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075759.592888:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075759.592898:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616441472:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075759.592899:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075759.592905:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075759.592906:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075759.592910:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2745 00000100:00080000:1.0:1769075759.592913:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075759.592915:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075759.592918:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075759.592925:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075759.592930:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616442112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075759.592951:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075759.592957:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1769075759.592958:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075759.592961:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075759.592968:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2745 00000100:00080000:1.0:1769075759.592973:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075759.592979:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075759/real 1769075759] req@ffff995b868c6300 x1855007616441600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075759.592980:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075759.592987:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075759.592998:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616442240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075759.593026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616441600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075759.593038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075759.593042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075759.593046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075759.593054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075759.593076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075759.593096:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075759/real 1769075759] req@ffff995b868c6d80 x1855007616441728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075759.593147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616441728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075759.593158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075759.593161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075759.593163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075759.593168:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441856:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075759.593192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075759.593204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075759/real 1769075759] req@ffff995b868c5c00 x1855007616441856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075759.593218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616441856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075759.593228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075759.593231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075759.593235:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075759.593240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441984:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075759.593255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075759.593263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075759/real 1769075759] req@ffff995b868c4a80 x1855007616441984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075759.593270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616441984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075759.593281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075759.593283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075759.593285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616441984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075759.593298:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616442112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075759.593309:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075759.593315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075759/real 1769075759] req@ffff995b868c7800 x1855007616442112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075759.593322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616442112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075759.593328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075759.593329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075759.593331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616442112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075759.593335:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616442240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075759.593344:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075759.593350:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075759/real 1769075759] req@ffff995b868c5500 x1855007616442240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075759.593357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616442240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075759.593362:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075759.593364:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075759.593366:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616442240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075760.296540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075760.296590:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075760.616530:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075760.616530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075760.616534:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075760.616588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075761.320527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075761.320607:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075761.640540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075761.640542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075761.640549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075761.640576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075762.344525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075762.344554:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075762.664548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075762.664548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075762.664567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075762.664582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075763.368523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075763.368580:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075763.688537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075763.688538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075763.688541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075763.688585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075764.392533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075764.392599:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075764.712537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075764.712537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075764.712539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075764.712561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769075764.712585:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075764.712590:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075764.712594:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075764.712601:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2750 00000100:00080000:0.0:1769075764.712610:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075764.712616:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075764.712648:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616442368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075764.712668:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075764.712671:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075764.712673:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075764.712676:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2750 00000100:00080000:0.0:1769075764.712678:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075764.712682:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075764.712690:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075764.712706:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616442368:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075764.712707:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616442496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075764.712724:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075764.712728:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075764.712730:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075764.712734:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2750 00000100:00080000:0.0:1769075764.712738:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075764.712742:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075764.712748:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075764.712760:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616442624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075764.712767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075764.712770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075764.712771:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075764.712774:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2750 00000100:00080000:0.0:1769075764.712777:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075764.712779:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769075764.712784:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769075764.712787:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075764.712797:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616442752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075764.712804:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075764.712806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075764.712807:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075764.712809:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2750 00000100:00080000:0.0:1769075764.712812:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075764.712814:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075764.712819:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075764.712825:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075764/real 1769075764] req@ffff995b83505180 x1855007616442368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075764.712833:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616442880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075764.712844:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075764.712845:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616442368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075764.712854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075764.712859:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075764.712868:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2750 00000100:00080000:2.0:1769075764.712869:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075764.712873:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075764.712878:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769075764.712879:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075764.712887:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075764.712889:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616442368:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075764.712899:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616443008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075764.712906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075764.712908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075764.712910:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075764.712913:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2750 00000100:00080000:0.0:1769075764.712916:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075764.712921:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616442496:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769075764.712922:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075764.712929:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075764.712939:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616443136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075764.712953:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075764.712967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075764/real 1769075764] req@ffff995b83504000 x1855007616442496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075764.712976:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616442496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075764.712985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075764.712989:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075764.712992:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616442496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075764.712999:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616442624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075764.713052:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075764.713063:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075764/real 1769075764] req@ffff995b83507100 x1855007616442624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075764.713071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616442624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075764.713079:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075764.713081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075764.713083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616442624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075764.713089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616442752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075764.713110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075764.713122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075764/real 1769075764] req@ffff995b83506300 x1855007616442752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075764.713136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616442752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075764.713146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075764.713149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075764.713152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616442752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075764.713157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616442880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075764.713171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075764.713179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075764/real 1769075764] req@ffff995b83507800 x1855007616442880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075764.713187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616442880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075764.713193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075764.713195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075764.713198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616442880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075764.713201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616443008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075764.713212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075764.713217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075764/real 1769075764] req@ffff995b83504e00 x1855007616443008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075764.713224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616443008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075764.713230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075764.713232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075764.713233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616443008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075764.713247:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616443136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075764.713257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075764.713266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075764/real 1769075764] req@ffff995b83505180 x1855007616443136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075764.713278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616443136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075764.713286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075764.713289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075764.713293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616443136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075765.416525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075765.416572:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075765.736536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075765.736537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075765.736562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075765.736588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075766.440535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075766.440564:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075766.760538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075766.760538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075766.760542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075766.760553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075767.464567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075767.464629:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075767.784529:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075767.784531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075767.784540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075767.784554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075768.488528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075768.488582:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075768.808564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075768.808564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075768.808564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075768.808595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075769.512528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075769.512571:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075769.832546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075769.832578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075769.832600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075769.832662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075769.832750:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075769.832757:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075769.832776:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075769.832797:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2755 00000100:00080000:2.0:1769075769.832836:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075769.832842:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075769.832875:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616443264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075769.832896:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075769.832899:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075769.832901:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075769.832904:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2755 00000100:00080000:2.0:1769075769.832907:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075769.832909:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075769.832917:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075769.832926:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616443392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075769.832947:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616443264:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075769.832948:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075769.832956:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075769.832958:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075769.832961:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2755 00000100:00080000:2.0:1769075769.832964:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075769.832967:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075769.832972:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075769.832984:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616443520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075769.832991:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075769.832993:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075769.832994:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075769.832997:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2755 00000100:00080000:2.0:1769075769.832999:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075769.833002:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075769.833005:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769075769.833011:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075769.833015:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616443648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075769.833023:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075769.833026:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075769.833027:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075769.833030:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2755 00000100:00080000:2.0:1769075769.833032:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075769.833041:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075769.833045:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075769.833051:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075769/real 1769075769] req@ffff995b85acbb80 x1855007616443264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075769.833055:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616443776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075769.833078:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075769.833080:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075769.833081:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075769.833085:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2755 00000100:00080000:2.0:1769075769.833087:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075769.833092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616443264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075769.833092:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075769.833099:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075769.833110:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616443904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075769.833114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075769.833119:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075769.833122:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075769.833124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769075769.833125:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075769.833132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616443264:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075769.833133:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2755 00000100:00080000:2.0:1769075769.833139:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075769.833143:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075769.833146:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075769.833153:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616444032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075769.833160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616443392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075769.833183:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075769.833193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075769/real 1769075769] req@ffff995b85ac8a80 x1855007616443392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.833202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616443392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.833208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.833211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075769.833213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616443392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075769.833217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616443520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075769.833228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075769.833244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075769/real 1769075769] req@ffff995b85aca300 x1855007616443520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.833257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616443520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.833264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.833266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075769.833269:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616443520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075769.833273:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616443648:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075769.833284:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075769.833290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075769/real 1769075769] req@ffff995b85acaa00 x1855007616443648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.833316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616443648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.833326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.833332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075769.833336:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616443648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075769.833344:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616443776:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075769.833366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075769.833378:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075769/real 1769075769] req@ffff995b85ac9c00 x1855007616443776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.833390:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616443776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.833400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.833403:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075769.833405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616443776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075769.833415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616443904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075769.833433:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075769.833440:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075769/real 1769075769] req@ffff995b85acb100 x1855007616443904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.833447:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616443904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.833471:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.833473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075769.833476:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616443904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075769.833491:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616444032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075769.833504:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075769.833511:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075769/real 1769075769] req@ffff995b85acb800 x1855007616444032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.833519:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616444032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.833524:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.833526:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075769.833528:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616444032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075770.536581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075770.536588:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075770.856617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075770.856617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075770.856666:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075770.856672:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075771.560553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075771.560649:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075771.880584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075771.880600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075771.880613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075771.880613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075772.584553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075772.584592:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075772.904555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075772.904562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075772.904562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075772.904574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075773.608542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075773.608596:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075773.928549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075773.928600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075773.928624:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075773.928647:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075774.632524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075774.632607:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075774.952579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075774.952589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075774.952615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075774.952661:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075774.952662:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075774.952671:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075774.952675:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075774.952687:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2760 00000100:00080000:3.0:1769075774.952697:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075774.952706:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075774.952736:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616444160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075774.952761:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075774.952764:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075774.952766:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075774.952770:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2760 00000100:00080000:3.0:1769075774.952774:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075774.952778:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075774.952788:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075774.952801:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616444288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075774.952802:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616444160:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075774.952818:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075774.952821:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075774.952823:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075774.952834:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2760 00000100:00080000:3.0:1769075774.952839:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075774.952842:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075774.952865:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075774.952879:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616444416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075774.952889:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769075774.952892:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075774.952892:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075774.952898:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075774.952906:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2760 00000100:00100000:0.0:1769075774.952967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075774/real 1769075774] req@ffff995b84559880 x1855007616444160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075774.952991:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616444160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075774.952995:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075774.953004:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769075774.953007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075774.953014:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769075774.953016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075774.953023:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616444160:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075774.953036:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616444544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075774.953049:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075774.953053:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075774.953056:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075774.953058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616444288:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075774.953065:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2760 00000100:00080000:3.0:1769075774.953075:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075774.953081:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769075774.953088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075774.953098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075774/real 1769075774] req@ffff995b84558700 x1855007616444288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075774.953106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616444288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075774.953112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075774.953114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075774.953117:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616444288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075774.953122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616444416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075774.953134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075774.953140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075774/real 1769075774] req@ffff995b8455a300 x1855007616444416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075774.953147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616444416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075774.953154:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075774.953156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075774.953159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616444416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075774.953163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616444544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075774.953177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075774.953184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075774/real 1769075774] req@ffff995b8455b100 x1855007616444544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075774.953192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616444544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075774.953197:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769075774.953199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075774.953204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075774.953207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616444544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769075774.953217:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616444672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075774.953276:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075774.953281:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075774.953283:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075774.953287:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2760 00000100:00080000:3.0:1769075774.953291:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075774.953293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616444672:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769075774.953298:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075774.953305:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075774.953316:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616444800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075774.953321:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075774.953332:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075774.953335:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769075774.953336:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075774/real 1769075774] req@ffff995b8455ad80 x1855007616444672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075774.953337:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075774.953346:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2760 00000100:00080000:3.0:1769075774.953353:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075774.953358:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075774.953359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616444672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075774.953362:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769075774.953372:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769075774.953377:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616444928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075774.953380:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075774.953387:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616444672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075774.953407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616444800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075774.953421:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075774.953428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075774/real 1769075774] req@ffff995b84559c00 x1855007616444800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075774.953440:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616444800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075774.953445:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075774.953447:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075774.953448:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616444800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075774.953470:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616444928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075774.953489:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075774.953499:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075774/real 1769075774] req@ffff995b8455b100 x1855007616444928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075774.953509:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616444928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075774.953516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075774.953518:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075774.953520:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616444928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075775.656535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075775.656620:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075775.976559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075775.976603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075775.976649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075775.976656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075776.680580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075776.680786:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075777.000587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075777.000611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075777.000644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075777.000690:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075777.704672:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075777.704699:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075778.024569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075778.024571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075778.024624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075778.024676:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075778.728597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075778.728715:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075779.048558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075779.048628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075779.048632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075779.048636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075779.752553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075779.752603:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075780.072571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075780.072599:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075780.072607:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075780.072613:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075780.072619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769075780.072626:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2765 00000100:00100000:2.0:1769075780.072632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769075780.072639:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075780.072649:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075780.072654:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075780.072692:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616445056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075780.072717:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075780.072720:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075780.072722:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075780.072725:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2765 00000100:00080000:0.0:1769075780.072727:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075780.072731:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075780.072739:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075780.072789:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616445056:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075780.072792:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616445184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075780.072814:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075780.072817:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075780.072819:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075780.072825:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2765 00000100:00080000:0.0:1769075780.072830:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075780.072834:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075780.072840:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075780.072852:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616445312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075780.072864:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075780.072867:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075780.072868:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075780.072870:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2765 00000100:00080000:0.0:1769075780.072885:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075780.072888:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075780.072892:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769075780.072898:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075780.072903:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616445440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075780.072911:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075780.072914:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075780.072916:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075780.072922:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2765 00000100:00080000:0.0:1769075780.072929:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075780.072932:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075780.072935:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075780.072944:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616445568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075780.072954:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075780.072958:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075780.072960:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075780.072965:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2765 00000100:00100000:1.0:1769075780.072966:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075780/real 1769075780] req@ffff995b83505180 x1855007616445056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075780.072974:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075780.072989:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075780.072999:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075780.073008:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616445056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075780.073022:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616445696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075780.073026:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075780.073040:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075780.073041:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769075780.073053:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769075780.073056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616445056:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769075780.073063:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075780.073069:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2765 00000100:00080000:0.0:1769075780.073074:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075780.073080:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075780.073085:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075780.073101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616445184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075780.073102:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616445824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075780.073150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075780.073166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075780/real 1769075780] req@ffff995b83504e00 x1855007616445184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075780.073179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616445184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075780.073187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075780.073191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075780.073194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616445184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075780.073200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616445312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075780.073214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075780.073222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075780/real 1769075780] req@ffff995b83507800 x1855007616445312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075780.073232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616445312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075780.073242:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075780.073245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075780.073247:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616445312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075780.073252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616445440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075780.073268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075780.073275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075780/real 1769075780] req@ffff995b83506300 x1855007616445440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075780.073286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616445440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075780.073295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075780.073297:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075780.073300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616445440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075780.073305:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616445568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075780.073318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075780.073326:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075780/real 1769075780] req@ffff995b83507100 x1855007616445568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075780.073336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616445568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075780.073345:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075780.073347:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075780.073350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616445568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075780.073355:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616445696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075780.073368:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075780.073390:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075780/real 1769075780] req@ffff995b83504000 x1855007616445696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075780.073408:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616445696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075780.073416:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075780.073419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075780.073422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616445696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075780.073446:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616445824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075780.073494:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075780.073505:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075780/real 1769075780] req@ffff995b83505180 x1855007616445824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075780.073517:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616445824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075780.073525:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075780.073528:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075780.073531:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616445824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075780.776568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075780.776664:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075781.096568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075781.096638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075781.096638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075781.096648:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075781.800622:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075781.800791:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075782.120651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075782.120651:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075782.120651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075782.120690:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075782.824610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075782.824790:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075783.144581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075783.144595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075783.144640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075783.144661:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075783.848601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075783.848743:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075784.168534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075784.168572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075784.168651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075784.168760:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075784.872657:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075784.872789:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075785.192564:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075785.192579:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769075785.192583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769075785.192585:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075785.192603:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2770 00000100:00080000:0.0:1769075785.192622:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075785.192635:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075785.192661:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075785.192724:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616445952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075785.192731:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075785.192741:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075785.192757:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075785.192763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075785.192767:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075785.192773:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2770 00000100:00080000:0.0:1769075785.192779:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075785.192785:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075785.192798:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075785.192810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616445952:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075785.192815:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616446080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075785.192836:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075785.192841:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075785.192844:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075785.192853:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2770 00000100:00080000:0.0:1769075785.192859:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075785.192864:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075785.192872:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075785.192889:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616446208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075785.192904:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075785.192909:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769075785.192918:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075785.192922:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075785.192929:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2770 00000100:00080000:0.0:1769075785.192939:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075785.192943:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075785.192951:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075785.192972:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616446336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075785.192974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075785/real 1769075785] req@ffff995b83505180 x1855007616445952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075785.193033:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075785.193044:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075785.193046:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075785.193053:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2770 00000100:00100000:1.0:1769075785.193058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616445952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075785.193067:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075785.193074:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075785.193076:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075785.193086:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769075785.193086:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075785.193103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616445952:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075785.193110:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616446464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075785.193132:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075785.193135:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075785.193138:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075785.193143:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2770 00000100:00080000:0.0:1769075785.193145:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075785.193149:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075785.193156:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075785.193163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075785.193172:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616446592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075785.193188:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075785.193192:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075785.193193:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075785.193199:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2770 00000100:00080000:0.0:1769075785.193206:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075785.193206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769075785.193221:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075785.193230:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075785.193242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075785/real 1769075785] req@ffff995b83504000 x1855007616446080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075785.193250:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616446720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075785.193325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616446080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075785.193342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075785.193350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075785.193356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075785.193369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075785.193425:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075785.193442:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075785/real 1769075785] req@ffff995b83507100 x1855007616446208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075785.193467:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616446208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075785.193514:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075785.193519:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075785.193524:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075785.193532:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446336:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075785.193558:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075785.193572:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075785/real 1769075785] req@ffff995b83506300 x1855007616446336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075785.193587:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616446336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075785.193598:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075785.193602:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075785.193606:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075785.193613:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446464:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075785.193630:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075785.193658:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075785/real 1769075785] req@ffff995b83507800 x1855007616446464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075785.193670:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616446464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075785.193714:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075785.193719:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075785.193722:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075785.193745:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616446592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075785.193770:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075785.193782:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075785/real 1769075785] req@ffff995b83504e00 x1855007616446592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075785.193795:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616446592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075785.193804:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075785.193807:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075785.193811:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616446592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075785.193817:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616446720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075785.193832:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075785.193841:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075785/real 1769075785] req@ffff995b83505180 x1855007616446720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075785.193853:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616446720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075785.193862:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075785.193865:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075785.193869:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616446720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075785.896552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075785.896629:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075786.216558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075786.216596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075786.216631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075786.216679:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075786.920555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075786.920660:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075787.240669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075787.240672:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075787.240713:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075787.240730:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075787.944574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075787.944745:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075788.264640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075788.264640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075788.264677:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075788.264683:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075788.968659:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075788.968745:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075789.288660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075789.288681:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075789.288681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075789.288724:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075789.992541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075789.992612:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075790.312582:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075790.312602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075790.312605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075790.312620:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075790.312626:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075790.312639:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2776 00000100:00080000:0.0:1769075790.312655:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075790.312665:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075790.312675:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075790.312678:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075790.312721:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616446848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075790.312750:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075790.312754:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075790.312756:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075790.312762:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2776 00000100:00080000:0.0:1769075790.312767:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075790.312771:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075790.312782:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075790.312789:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616446848:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075790.312795:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616446976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075790.312804:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075790.312807:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075790.312808:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075790.312811:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2776 00000100:00080000:0.0:1769075790.312815:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075790.312818:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075790.312821:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075790.312829:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616447104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075790.312837:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075790.312840:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075790.312841:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075790.312844:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2776 00000100:00080000:0.0:1769075790.312862:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075790.312865:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075790.312868:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769075790.312875:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075790.312880:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616447232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075790.312894:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075790.312897:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075790.312899:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075790.312902:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2776 00000100:00080000:0.0:1769075790.312904:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075790.312907:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075790.312910:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075790.312919:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616447360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075790.312927:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075790/real 1769075790] req@ffff995b83505180 x1855007616446848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075790.312929:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075790.312936:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075790.312939:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075790.312946:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2776 00000100:00080000:0.0:1769075790.312950:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075790.312953:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075790.312956:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075790.312958:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616446848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075790.312969:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616447488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075790.312970:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075790.312979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769075790.312986:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075790.312987:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616446848:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769075790.312992:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075790.312995:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075790.313000:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2776 00000100:00080000:0.0:1769075790.313002:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075790.313005:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075790.313008:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075790.313016:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616447616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075790.313018:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075790.313047:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075790.313064:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075790/real 1769075790] req@ffff995b83504e00 x1855007616446976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075790.313086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616446976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075790.313097:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075790.313101:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075790.313105:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616446976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075790.313111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616447104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075790.313131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075790.313139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075790/real 1769075790] req@ffff995b83507800 x1855007616447104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075790.313148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616447104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075790.313156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075790.313159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075790.313161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616447104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075790.313166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616447232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075790.313178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075790.313184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075790/real 1769075790] req@ffff995b83506300 x1855007616447232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075790.313192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616447232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075790.313203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075790.313205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075790.313207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616447232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075790.313211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616447360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075790.313220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075790.313226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075790/real 1769075790] req@ffff995b83507100 x1855007616447360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075790.313234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616447360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075790.313241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075790.313242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075790.313244:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616447360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075790.313247:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616447488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075790.313257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075790.313263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075790/real 1769075790] req@ffff995b83504000 x1855007616447488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075790.313270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616447488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075790.313275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075790.313277:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075790.313280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616447488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075790.313294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616447616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075790.313305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075790.313312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075790/real 1769075790] req@ffff995b83505180 x1855007616447616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075790.313319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616447616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075790.313325:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075790.313326:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075790.313328:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616447616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075791.016686:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075791.016757:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075791.336623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075791.336647:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075791.336684:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075791.336698:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075792.040588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075792.040770:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075792.360616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075792.360658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075792.360663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075792.360718:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075793.064637:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075793.064743:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075793.384663:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075793.384663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075793.384682:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075793.384711:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075794.088601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075794.088725:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075794.408572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075794.408576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075794.408576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075794.408577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075795.112520:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075795.112599:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075795.432523:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075795.432545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075795.432553:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075795.432559:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769075795.432563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075795.432564:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075795.432574:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2781 00000100:00080000:1.0:1769075795.432585:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075795.432587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075795.432592:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075795.432632:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616447744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075795.432669:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075795.432673:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075795.432675:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075795.432678:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2781 00000100:00080000:1.0:1769075795.432682:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075795.432687:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075795.432694:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075795.432699:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616447744:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075795.432709:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616447872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075795.432718:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075795.432721:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075795.432722:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075795.432725:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2781 00000100:00080000:1.0:1769075795.432727:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075795.432730:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075795.432733:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075795.432741:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616448000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075795.432746:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075795.432749:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075795.432751:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075795.432753:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2781 00000100:00080000:1.0:1769075795.432756:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075795.432760:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769075795.432764:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769075795.432766:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075795.432778:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616448128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075795.432784:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075795.432797:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075795.432799:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075795.432801:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2781 00000100:00080000:1.0:1769075795.432803:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075795.432806:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075795.432810:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075795.432813:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075795/real 1769075795] req@ffff995b868c7480 x1855007616447744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075795.432821:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616448256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075795.432829:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075795.432833:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616447744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075795.432833:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075795.432837:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075795.432845:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2781 00000100:00080000:1.0:1769075795.432849:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075795.432851:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075795.432864:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769075795.432868:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769075795.432870:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075795.432874:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616447744:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075795.432881:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616448384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075795.432894:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075795.432897:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616447872:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075795.432898:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075795.432903:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075795.432908:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2781 00000100:00080000:1.0:1769075795.432911:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075795.432913:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769075795.432920:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769075795.432928:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075795.432938:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075795/real 1769075795] req@ffff995b868c4700 x1855007616447872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075795.432942:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616448512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075795.432961:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616447872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075795.432969:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075795.432972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075795.432976:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616447872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075795.432981:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075795.432992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075795.432998:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075795/real 1769075795] req@ffff995b868c6a00 x1855007616448000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075795.433005:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616448000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075795.433024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075795.433026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075795.433028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075795.433031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448128:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075795.433041:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075795.433047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075795/real 1769075795] req@ffff995b868c5500 x1855007616448128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075795.433054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616448128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075795.433064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075795.433066:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075795.433068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075795.433071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075795.433080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075795.433086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075795/real 1769075795] req@ffff995b868c7800 x1855007616448256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075795.433093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616448256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075795.433098:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075795.433100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075795.433102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075795.433112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616448384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075795.433122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075795.433129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075795/real 1769075795] req@ffff995b868c4a80 x1855007616448384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075795.433136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616448384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075795.433141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075795.433143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075795.433145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616448384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075795.433148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616448512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075795.433157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075795.433163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075795/real 1769075795] req@ffff995b868c5c00 x1855007616448512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075795.433169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616448512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075795.433176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075795.433178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075795.433180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616448512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075796.136603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075796.136711:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075796.456564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075796.456565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075796.456596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075796.456614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075797.160550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075797.160688:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075797.480616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075797.480707:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075797.480708:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075797.480749:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075798.184545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075798.184673:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075798.504628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075798.504628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075798.504631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075798.504667:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075799.208620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075799.208742:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075799.528599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075799.528606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075799.528693:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075799.528707:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075800.232629:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075800.232674:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075800.552553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075800.552565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075800.552574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075800.552594:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769075800.552594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769075800.552601:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075800.552605:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075800.552612:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2786 00000100:00080000:2.0:1769075800.552622:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075800.552628:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075800.552659:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616448640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075800.552682:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075800.552685:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075800.552686:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075800.552689:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2786 00000100:00080000:2.0:1769075800.552692:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075800.552695:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075800.552702:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075800.552721:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616448768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075800.552725:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616448640:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075800.552736:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075800.552739:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075800.552740:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075800.552743:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2786 00000100:00080000:2.0:1769075800.552745:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075800.552748:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075800.552751:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075800.552760:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616448896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075800.552766:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075800.552768:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075800.552770:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075800.552772:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2786 00000100:00080000:2.0:1769075800.552774:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075800.552776:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075800.552779:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075800.552787:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616449024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075800.552802:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075800.552805:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075800.552806:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075800.552809:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2786 00000100:00080000:2.0:1769075800.552812:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075800.552814:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075800.552817:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075800.552828:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616449152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075800.552834:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075800.552836:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769075800.552840:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448768:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075800.552840:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075800.552848:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2786 00000100:00080000:2.0:1769075800.552852:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075800.552855:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075800.552859:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075800.552868:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616449280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075800.552876:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075800.552879:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075800.552880:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075800.552883:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2786 00000100:00080000:2.0:1769075800.552887:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075800.552890:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448896:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075800.552891:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075800.552897:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075800.552908:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616449408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075800.552929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075800.552946:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075800.552964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616449280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075800.552977:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616449408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075801.256574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075801.256689:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075801.576541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075801.576541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075801.576563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075801.576723:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075802.280624:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075802.280707:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075802.600594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075802.600612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075802.600619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075802.600720:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075803.304537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075803.304572:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075803.624545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075803.624549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075803.624610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075803.624800:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:1.0:1769075803.624816:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769075803.624885:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769075803.624903:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075803.624964:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075803.624978:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075803.624989:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075803.624999:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075803.625009:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075803.625018:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769075803.625023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075800/real 1769075803] req@ffff995b85ac8380 x1855007616448768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075803.625069:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616448768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075803.625084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075803.625091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075803.625096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075803.625106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075800/real 1769075803] req@ffff995b85acb800 x1855007616448896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075803.625120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616448896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075803.625132:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075803.625136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075803.625140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616448896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075803.625151:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075800/real 1769075803] req@ffff995b85acb100 x1855007616449024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075803.625166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616449024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075803.625176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075803.625180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075803.625184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075803.625191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075800/real 1769075803] req@ffff995b85ac9c00 x1855007616449152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075803.625203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616449152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075803.625213:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075803.625216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075803.625220:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075803.625227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075800/real 1769075803] req@ffff995b85ac9880 x1855007616448640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075803.625240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616448640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075803.625248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075803.625253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075803.625256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616448640:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075803.625294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075800/real 1769075803] req@ffff995b85acaa00 x1855007616449280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075803.625309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616449280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075803.625318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075803.625322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075803.625326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616449280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075803.625333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075800/real 1769075803] req@ffff995b85aca300 x1855007616449408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075803.625346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616449408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075803.625353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075803.625357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075803.625361:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616449408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075804.328576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075804.328662:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075804.648602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075804.648642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075804.648680:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075804.648689:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075805.352527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075805.352639:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075805.672649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075805.672658:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075805.672672:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075805.672681:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075805.672692:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075805.672698:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2791 00000100:00080000:0.0:1769075805.672744:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075805.672750:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769075805.672761:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075805.672812:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075805.672839:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616449536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075805.672920:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075805.672929:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075805.672932:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075805.672941:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2791 00000100:00080000:0.0:1769075805.672948:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075805.672972:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075805.672987:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075805.673011:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616449664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075805.673034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616449536:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769075805.673035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075805.673052:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075805.673056:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075805.673064:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2791 00000100:00080000:0.0:1769075805.673074:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075805.673081:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075805.673092:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075805.673116:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616449792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075805.673133:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075805.673140:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075805.673144:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075805.673150:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2791 00000100:00080000:0.0:1769075805.673158:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075805.673177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769075805.673180:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075805.673193:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075805.673214:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616449920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075805.673230:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075805.673235:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075805.673239:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075805.673248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075805/real 1769075805] req@ffff995b83505180 x1855007616449536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075805.673249:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2791 00000100:00080000:0.0:1769075805.673260:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075805.673281:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075805.673289:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075805.673298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616449536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075805.673315:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616450048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075805.673328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075805.673363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075805.673370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769075805.673374:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075805.673380:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075805.673385:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616449536:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769075805.673392:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2791 00000100:00080000:0.0:1769075805.673405:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075805.673412:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075805.673420:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075805.673442:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075805.673489:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616450176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075805.673510:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075805.673517:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075805.673522:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075805.673530:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2791 00000100:00080000:0.0:1769075805.673538:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075805.673544:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075805.673555:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075805.673582:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616450304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075805.673590:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075805.673631:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075805/real 1769075805] req@ffff995b83504000 x1855007616449664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075805.673666:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616449664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075805.673683:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075805.673689:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075805.673704:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075805.673721:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075805.673759:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075805.673779:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075805/real 1769075805] req@ffff995b83507100 x1855007616449792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075805.673800:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616449792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075805.673819:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075805.673825:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075805.673830:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075805.673841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075805.673875:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075805.673895:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075805/real 1769075805] req@ffff995b83506300 x1855007616449920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075805.673916:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616449920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075805.673932:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075805.673937:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075805.673943:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616449920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075805.673952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075805.673985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075805.674004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075805/real 1769075805] req@ffff995b83507800 x1855007616450048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075805.674037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616450048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075805.674052:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075805.674057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075805.674061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075805.674093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616450176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075805.674126:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075805.674145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075805/real 1769075805] req@ffff995b83504e00 x1855007616450176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075805.674167:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616450176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075805.674180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075805.674185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075805.674191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616450176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075805.674200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616450304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075805.674236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075805.674259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075805/real 1769075805] req@ffff995b83505180 x1855007616450304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075805.674280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616450304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075805.674306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075805.674312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075805.674317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616450304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075806.376568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075806.376659:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075806.696591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075806.696592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075806.696678:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075806.696698:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075807.400547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075807.400741:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075807.720576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075807.720591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075807.720633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075807.720638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075808.424552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075808.424633:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075808.744594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075808.744616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075808.744631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075808.744643:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075809.448559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075809.448613:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075809.768543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075809.768544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075809.768544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075809.768650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075810.472546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075810.472608:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075810.792514:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075810.792553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075810.792568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075810.792570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075810.792593:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075810.792598:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075810.792601:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075810.792611:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2796 00000100:00080000:1.0:1769075810.792620:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075810.792627:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075810.792657:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616450432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075810.792680:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075810.792683:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075810.792684:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075810.792687:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2796 00000100:00080000:1.0:1769075810.792690:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075810.792694:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075810.792702:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075810.792712:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616450432:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075810.792715:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616450560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075810.792726:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075810.792729:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075810.792731:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075810.792735:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2796 00000100:00080000:1.0:1769075810.792739:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075810.792741:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075810.792745:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075810.792755:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616450688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075810.792763:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075810.792766:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075810.792767:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075810.792771:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2796 00000100:00080000:1.0:1769075810.792775:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075810.792777:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075810.792782:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769075810.792786:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075810.792793:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616450816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075810.792813:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075810.792815:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075810.792817:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075810.792820:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2796 00000100:00080000:1.0:1769075810.792822:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075810.792824:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075810.792828:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075810.792834:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616450944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075810.792840:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075810.792842:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075810.792843:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075810.792846:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2796 00000100:00080000:1.0:1769075810.792847:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075810.792850:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075810.792853:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075810.792858:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075810/real 1769075810] req@ffff995b868c6d80 x1855007616450432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075810.792861:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616451072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075810.792873:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075810.792882:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075810.792884:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075810.792887:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2796 00000100:00080000:1.0:1769075810.792891:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075810.792897:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075810.792901:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075810.792908:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616451200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075810.792912:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616450432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075810.792927:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075810.792933:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075810.792939:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616450432:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075810.792963:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075810.792993:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075810.793014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075810/real 1769075810] req@ffff995b868c6300 x1855007616450560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075810.793026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616450560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075810.793033:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075810.793037:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075810.793041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075810.793048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075810.793069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075810.793086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075810/real 1769075810] req@ffff995b868c7100 x1855007616450688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075810.793102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616450688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075810.793113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075810.793117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075810.793121:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075810.793128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450816:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075810.793167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075810.793178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075810/real 1769075810] req@ffff995b868c5c00 x1855007616450816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075810.793188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616450816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075810.793195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075810.793197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075810.793200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075810.793204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450944:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075810.793217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075810.793224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075810/real 1769075810] req@ffff995b868c4a80 x1855007616450944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075810.793232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616450944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075810.793239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075810.793241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075810.793243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616450944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075810.793266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616451072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075810.793283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075810.793308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075810/real 1769075810] req@ffff995b868c7800 x1855007616451072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075810.793316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616451072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075810.793323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075810.793325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075810.793327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616451072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075810.793331:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616451200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075810.793342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075810.793347:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075810/real 1769075810] req@ffff995b868c5500 x1855007616451200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075810.793355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616451200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075810.793360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075810.793362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075810.793364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616451200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075811.496546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075811.496717:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075811.816611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075811.816630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075811.816647:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075811.816686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075812.520582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075812.520695:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075812.840584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075812.840614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075812.840689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075812.840705:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075813.544567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075813.544682:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075813.864510:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075813.864546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075813.864554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075813.864557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075814.568569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075814.568734:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075814.888581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075814.888604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075814.888607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075814.888620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075815.592510:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075815.592863:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075815.912616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075815.912620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075815.912697:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075815.913584:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075815.913594:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769075815.913594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769075815.913610:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075815.913628:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2801 00000100:00080000:0.0:1769075815.913644:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075815.913671:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075815.913718:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616451328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075815.913750:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075815.913764:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075815.913766:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075815.913770:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2801 00000100:00080000:0.0:1769075815.913773:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075815.913777:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075815.913787:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075815.913800:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616451456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075815.913810:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075815.913816:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075815.913818:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075815.913828:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616451328:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769075815.913832:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2801 00000100:00080000:0.0:1769075815.913840:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075815.913862:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075815.913867:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075815.913882:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616451584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075815.913893:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075815.913897:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075815.913899:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075815.913904:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2801 00000100:00080000:0.0:1769075815.913909:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075815.913912:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075815.913916:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075815.913928:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616451712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075815.913938:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075815.913944:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769075815.913946:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075815.913950:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075815.913956:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2801 00000100:00080000:0.0:1769075815.913970:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075815.913977:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075815.913985:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075815.914005:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616451840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075815.914012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075815/real 1769075815] req@ffff995b83505180 x1855007616451328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075815.914065:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075815.914069:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075815.914072:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075815.914072:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616451328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075815.914083:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2801 00000100:00080000:0.0:1769075815.914091:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075815.914093:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075815.914098:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769075815.914103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769075815.914108:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075815.914110:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616451328:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075815.914127:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616451968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075815.914139:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075815.914142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616451456:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769075815.914144:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075815.914150:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075815.914156:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2801 00000100:00080000:0.0:1769075815.914178:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075815.914185:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769075815.914190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769075815.914225:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075815.914237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075815/real 1769075815] req@ffff995b83504e00 x1855007616451456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075815.914251:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616452096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075815.914257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616451456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075815.914296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075815.914305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075815.914310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616451456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075815.914318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616451584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075815.914354:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075815.914364:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075815/real 1769075815] req@ffff995b83507800 x1855007616451584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075815.914373:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616451584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075815.914393:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075815.914396:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075815.914399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616451584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075815.914404:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616451712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075815.914493:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075815.914520:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075815/real 1769075815] req@ffff995b83506300 x1855007616451712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075815.914530:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616451712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075815.914538:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075815.914541:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075815.914544:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616451712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075815.914549:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616451840:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075815.914576:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075815.914585:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075815/real 1769075815] req@ffff995b83507100 x1855007616451840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075815.914594:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616451840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075815.914603:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075815.914606:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075815.914608:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616451840:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075815.914624:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616451968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075815.914637:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075815.914644:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075815/real 1769075815] req@ffff995b83504000 x1855007616451968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075815.914653:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616451968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075815.914660:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075815.914662:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075815.914664:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616451968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075815.914668:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616452096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075815.914679:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075815.914686:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075815/real 1769075815] req@ffff995b83505180 x1855007616452096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075815.914695:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616452096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075815.914701:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075815.914703:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075815.914707:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616452096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075816.616535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075816.616627:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075816.936538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075816.936568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075816.936603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075816.937530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075817.640635:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075817.640726:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075817.960569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075817.960612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075817.960617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075817.960656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075818.664538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075818.664622:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075818.984607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075818.984610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075818.984665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075818.984669:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075819.688642:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075819.688702:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075820.008607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075820.008609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075820.008631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075820.008772:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075820.712608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075820.712712:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075821.032699:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075821.032713:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075821.032720:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075821.032742:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2806 00000100:00100000:0.0:1769075821.032756:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075821.032756:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075821.032759:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075821.032759:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075821.032789:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075821.032841:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075821.032853:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616452224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075821.032895:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075821.032922:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075821.032926:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075821.032936:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2806 00000100:00100000:2.0:1769075821.032947:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616452224:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075821.032949:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075821.032968:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075821.032982:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075821.033011:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616452352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075821.033029:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075821.033033:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075821.033036:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075821.033042:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2806 00000100:00080000:1.0:1769075821.033050:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075821.033054:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075821.033059:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075821.033101:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616452480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075821.033115:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075821.033119:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075821.033122:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075821.033127:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2806 00000800:00000100:2.0:1769075821.033142:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075821.033146:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075821.033158:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075821.033168:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075821.033193:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616452608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075821.033208:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075821.033212:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075821.033216:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075821.033221:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2806 00000100:00080000:1.0:1769075821.033227:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075821.033234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075821/real 1769075821] req@ffff995b868c6a00 x1855007616452224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075821.033241:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075821.033256:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075821.033272:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616452736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075821.033303:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075821.033308:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075821.033314:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075821.033315:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616452224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075821.033339:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2806 00000100:00080000:1.0:1769075821.033354:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075821.033360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075821.033363:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769075821.033380:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769075821.033382:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075821.033395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616452224:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075821.033419:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616452864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075821.033446:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616452352:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075821.033447:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075821.033525:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075821.033531:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769075821.033542:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075821.033551:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2806 00000100:00080000:1.0:1769075821.033566:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075821.033581:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075821.033583:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075821/real 1769075821] req@ffff995b868c4700 x1855007616452352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769075821.033593:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075821.033623:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616452992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075821.033639:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616452352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075821.033668:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075821.033678:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075821.033687:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616452352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075821.033702:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616452480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075821.033747:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075821.033766:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075821/real 1769075821] req@ffff995b868c7480 x1855007616452480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075821.033783:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616452480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075821.033798:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075821.033803:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075821.033807:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616452480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075821.033816:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616452608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075821.033840:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075821.033852:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075821/real 1769075821] req@ffff995b868c5500 x1855007616452608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075821.033866:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616452608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075821.033887:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075821.033891:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075821.033896:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616452608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075821.033902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616452736:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075821.033920:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075821.033932:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075821/real 1769075821] req@ffff995b868c7800 x1855007616452736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075821.033945:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616452736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075821.033957:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075821.033961:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075821.033965:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616452736:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075821.033990:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616452864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075821.034010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075821.034023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075821/real 1769075821] req@ffff995b868c4a80 x1855007616452864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075821.034038:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616452864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075821.034048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075821.034052:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075821.034057:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616452864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075821.034065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616452992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075821.034132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075821.034155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075821/real 1769075821] req@ffff995b868c5c00 x1855007616452992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075821.034173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616452992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075821.034184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075821.034188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075821.034192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616452992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075821.736671:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075821.736765:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075822.056617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075822.056689:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075822.056692:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075822.056718:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075822.760561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075822.760658:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075823.080600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075823.080602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075823.080667:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075823.080714:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075823.784648:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075823.784699:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075824.104709:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075824.104710:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075824.104710:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075824.104710:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075824.808543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075824.808679:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075825.128540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075825.128567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075825.128618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075825.128655:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075825.832550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075825.832665:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075826.152545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075826.152545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075826.152549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075826.152577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075826.152582:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075826.152587:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075826.152590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075826.152597:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2811 00000100:00080000:3.0:1769075826.152607:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075826.152613:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075826.152645:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616453120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075826.152673:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075826.152678:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075826.152680:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075826.152684:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2811 00000100:00080000:3.0:1769075826.152689:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075826.152695:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075826.152705:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616453120:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769075826.152706:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075826.152721:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616453248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075826.152730:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075826.152732:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075826.152734:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075826.152738:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2811 00000100:00080000:3.0:1769075826.152741:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075826.152743:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075826.152747:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075826.152755:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616453376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075826.152761:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075826.152763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075826.152764:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075826.152768:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2811 00000800:00000100:2.0:1769075826.152773:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075826.152774:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075826.152780:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075826.152784:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075826.152793:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616453504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075826.152799:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075826.152801:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075826.152803:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075826.152806:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2811 00000100:00080000:3.0:1769075826.152809:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075826.152813:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075826/real 1769075826] req@ffff995b8455a300 x1855007616453120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075826.152814:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075826.152821:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075826.152834:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616453632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075826.152843:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075826.152844:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616453120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075826.152849:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075826.152853:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075826.152862:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2811 00000100:00080000:2.0:1769075826.152863:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075826.152865:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075826.152870:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075826.152871:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075826.152896:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075826.152897:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616453120:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075826.152911:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616453760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075826.152920:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075826.152924:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075826.152926:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075826.152931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616453248:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075826.152932:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2811 00000100:00080000:3.0:1769075826.152942:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075826.152947:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075826.152951:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075826.152965:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616453888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075826.152974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075826.153018:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075826/real 1769075826] req@ffff995b84558700 x1855007616453248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075826.153035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616453248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075826.153044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075826.153047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075826.153050:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616453248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075826.153057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616453376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075826.153075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075826.153084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075826/real 1769075826] req@ffff995b84559880 x1855007616453376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075826.153095:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616453376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075826.153109:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075826.153112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075826.153115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616453376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075826.153120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616453504:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075826.153133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075826.153141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075826/real 1769075826] req@ffff995b8455b480 x1855007616453504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075826.153164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616453504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075826.153171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075826.153173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075826.153175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616453504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075826.153179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616453632:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075826.153189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075826.153195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075826/real 1769075826] req@ffff995b84558000 x1855007616453632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075826.153201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616453632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075826.153208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075826.153210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075826.153213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616453632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075826.153216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616453760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075826.153225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075826.153230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075826/real 1769075826] req@ffff995b8455a680 x1855007616453760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075826.153237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616453760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075826.153242:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075826.153244:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075826.153246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616453760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075826.153259:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616453888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075826.153268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075826.153274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075826/real 1769075826] req@ffff995b84558a80 x1855007616453888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075826.153282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616453888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075826.153287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075826.153289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075826.153291:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616453888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075826.856555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075826.856645:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075827.176603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075827.176612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075827.176617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075827.176679:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075827.880520:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075827.880587:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075828.200548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075828.200548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075828.200569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075828.200573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075828.904507:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075828.904613:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075829.224603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075829.224604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075829.224609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075829.224649:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075829.928558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075829.928678:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075830.248540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075830.248554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075830.248600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075830.248635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075830.952546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075830.952750:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075831.272589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075831.272674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075831.272676:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075831.272700:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769075831.272740:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075831.272749:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075831.272755:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075831.272767:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2816 00000100:00080000:2.0:1769075831.272806:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075831.272816:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075831.272862:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616454016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075831.272975:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075831.273001:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075831.273008:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075831.273018:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616454016:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075831.273030:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2816 00000100:00080000:2.0:1769075831.273043:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075831.273053:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075831.273069:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075831.273114:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616454144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075831.273140:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075831.273144:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075831.273147:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075831.273151:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2816 00000100:00080000:2.0:1769075831.273157:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075831.273161:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769075831.273184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769075831.273190:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075831.273212:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616454272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075831.273226:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075831.273231:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075831.273234:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075831.273239:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2816 00000100:00080000:2.0:1769075831.273245:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075831.273249:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075831.273257:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075831.273273:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616454400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075831.273287:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075831/real 1769075831] req@ffff995b85ac8a80 x1855007616454016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075831.273299:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075831.273348:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075831.273352:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075831.273361:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616454016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075831.273369:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2816 00000100:00080000:2.0:1769075831.273389:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075831.273408:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769075831.273412:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769075831.273417:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769075831.273425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075831.273445:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616454016:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075831.273459:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616454528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075831.273520:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075831.273522:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616454144:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075831.273536:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075831.273546:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075831.273553:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2816 00000100:00080000:2.0:1769075831.273558:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075831.273563:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075831.273573:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075831.273591:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616454656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075831.273603:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769075831.273614:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075831.273626:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075831.273633:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075831.273649:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2816 00000100:00100000:0.0:1769075831.273650:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075831/real 1769075831] req@ffff995b85acbb80 x1855007616454144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075831.273673:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075831.273685:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616454144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075831.273695:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075831.273725:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769075831.273726:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075831.273741:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075831.273748:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616454144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075831.273757:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616454784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075831.273776:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616454272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075831.273831:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075831.273853:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075831/real 1769075831] req@ffff995b85aca300 x1855007616454272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075831.273872:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616454272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075831.273887:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075831.273891:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075831.273895:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616454272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075831.273943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616454400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075831.273977:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075831.274002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075831/real 1769075831] req@ffff995b85acaa00 x1855007616454400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075831.274018:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616454400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075831.274030:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075831.274035:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075831.274039:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616454400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075831.274064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616454528:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075831.274084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075831.274096:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075831/real 1769075831] req@ffff995b85ac9880 x1855007616454528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075831.274118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616454528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075831.274139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075831.274146:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075831.274153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616454528:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075831.274166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616454656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075831.274215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075831.274238:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075831/real 1769075831] req@ffff995b85ac9c00 x1855007616454656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075831.274264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616454656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075831.274279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075831.274284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075831.274288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616454656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075831.274297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616454784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075831.274326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075831.274340:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075831/real 1769075831] req@ffff995b85acb100 x1855007616454784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075831.274356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616454784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075831.274367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075831.274371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075831.274376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616454784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075831.976550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075831.976591:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075832.296603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075832.296606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075832.296639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075832.296736:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075833.000575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075833.000670:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075833.320593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075833.320607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075833.320656:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075833.320677:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075834.024576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075834.024700:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075834.344615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075834.344616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075834.344626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075834.344644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075835.048632:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075835.048755:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075835.368582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075835.368665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075835.368677:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075835.368732:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075836.072573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075836.072657:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075836.392592:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075836.392613:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769075836.392615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075836.392618:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075836.392619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075836.392631:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2822 00000100:00100000:2.0:1769075836.392643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075836.392646:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075836.392657:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075836.392669:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075836.392701:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616454912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075836.392734:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075836.392737:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075836.392739:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075836.392743:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2822 00000100:00080000:3.0:1769075836.392747:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075836.392752:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075836.392762:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075836.392774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616454912:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075836.392778:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616455040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075836.392790:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075836.392793:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075836.392794:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075836.392798:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2822 00000100:00080000:3.0:1769075836.392801:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075836.392803:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075836.392806:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075836.392838:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616455168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075836.392846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075836.392849:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075836.392850:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075836.392853:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2822 00000100:00080000:3.0:1769075836.392857:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075836.392860:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075836.392865:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769075836.392879:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075836.392880:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616455296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075836.392891:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075836.392894:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075836.392896:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075836.392899:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2822 00000100:00080000:3.0:1769075836.392903:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075836.392905:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075836.392909:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075836.392920:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616455424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075836.392927:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075836/real 1769075836] req@ffff995b8455bb80 x1855007616454912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075836.392931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075836.392941:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075836.392945:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075836.392952:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2822 00000100:00080000:3.0:1769075836.392955:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075836.392960:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616454912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075836.392963:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075836.392971:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769075836.392980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769075836.392985:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616455552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075836.392992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075836.392999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075836.393005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616454912:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075836.393006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075836.393011:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075836.393019:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2822 00000100:00080000:3.0:1769075836.393023:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075836.393025:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075836.393028:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075836.393036:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616455680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075836.393049:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075836.393077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075836.393089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075836/real 1769075836] req@ffff995b8455aa00 x1855007616455040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075836.393099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616455040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075836.393106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075836.393109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075836.393112:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075836.393117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075836.393130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075836.393137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075836/real 1769075836] req@ffff995b84559500 x1855007616455168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075836.393146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616455168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075836.393153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075836.393155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075836.393158:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075836.393162:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455296:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075836.393174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075836.393180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075836/real 1769075836] req@ffff995b84558a80 x1855007616455296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075836.393189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616455296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075836.393195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075836.393197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075836.393199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075836.393203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455424:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075836.393214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075836.393221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075836/real 1769075836] req@ffff995b8455a680 x1855007616455424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075836.393230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616455424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075836.393236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075836.393238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075836.393241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455424:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075836.393244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616455552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075836.393255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075836.393261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075836/real 1769075836] req@ffff995b84558000 x1855007616455552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075836.393270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616455552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075836.393276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075836.393278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075836.393280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616455552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075836.393297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616455680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075836.393336:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075836.393355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075836/real 1769075836] req@ffff995b8455b480 x1855007616455680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075836.393403:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616455680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075836.393425:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075836.393429:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075836.393432:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616455680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075837.096513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075837.096663:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075837.416580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075837.416643:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075837.416643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075837.416689:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075838.120584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075838.120833:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075838.440624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075838.440659:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075838.440701:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075838.440702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075839.144565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075839.144659:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075839.464604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075839.464639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075839.464677:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075839.464704:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075840.168559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075840.168592:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075840.488553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075840.488565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075840.488579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075840.488592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075841.192543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075841.192647:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075841.512598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075841.512624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075841.512675:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769075841.512690:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075841.512699:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075841.512705:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075841.512707:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075841.512719:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2827 00000100:00080000:2.0:1769075841.512790:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075841.512802:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075841.512886:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616455808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075841.512927:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075841.512931:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075841.512934:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075841.512938:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2827 00000100:00080000:2.0:1769075841.512943:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075841.512948:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075841.512960:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075841.512976:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616455936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075841.512997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616455808:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075841.513023:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075841.513030:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075841.513034:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075841.513043:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2827 00000100:00080000:2.0:1769075841.513051:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075841.513058:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075841.513067:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769075841.513102:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075841.513106:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616456064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075841.513141:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075841.513148:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075841.513154:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075841.513155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075841/real 1769075841] req@ffff995b85acb800 x1855007616455808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075841.513170:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2827 00000100:00080000:2.0:1769075841.513185:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075841.513192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616455808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075841.513198:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769075841.513223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769075841.513223:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769075841.513236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075841.513242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616455808:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075841.513250:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616456192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075841.513263:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075841.513267:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075841.513270:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075841.513275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455936:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075841.513281:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2827 00000100:00080000:2.0:1769075841.513291:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075841.513296:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075841.513302:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769075841.513319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075841.513321:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616456320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075841.513339:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075841.513344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075841/real 1769075841] req@ffff995b85ac8380 x1855007616455936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075841.513346:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075841.513351:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075841.513362:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2827 00000100:00080000:2.0:1769075841.513371:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075841.513379:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075841.513385:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616455936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769075841.513388:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769075841.513402:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075841.513407:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075841.513411:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616455936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075841.513427:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075841.513435:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616456448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075841.513485:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075841.513489:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075841.513491:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769075841.513496:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075841.513503:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2827 00000100:00080000:2.0:1769075841.513511:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075841.513517:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075841.513524:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075841.513528:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075841/real 1769075841] req@ffff995b85acb100 x1855007616456064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075841.513557:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616456576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075841.513561:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616456064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075841.513606:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075841.513611:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075841.513616:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075841.513625:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075841.513648:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075841.513658:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075841/real 1769075841] req@ffff995b85ac9c00 x1855007616456192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075841.513678:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616456192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075841.513687:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075841.513690:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075841.513693:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075841.513709:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456320:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075841.513725:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075841.513772:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075841/real 1769075841] req@ffff995b85ac9880 x1855007616456320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075841.513785:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616456320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075841.513794:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075841.513798:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075841.513802:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456320:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075841.513808:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616456448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075841.513826:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075841.513840:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075841/real 1769075841] req@ffff995b85acaa00 x1855007616456448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075841.513852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616456448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075841.513867:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075841.513870:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075841.513873:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616456448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075841.513879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616456576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075841.513894:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075841.513904:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075841/real 1769075841] req@ffff995b85aca300 x1855007616456576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075841.513914:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616456576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075841.513923:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075841.513926:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075841.513930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616456576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075842.216572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075842.216687:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075842.536590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075842.536591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075842.536634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075842.536681:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075843.240542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075843.240637:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075843.560606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075843.560645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075843.560646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075843.560651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075844.264526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075844.264601:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075844.584556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075844.584564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075844.584564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075844.584567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075845.288605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075845.288763:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075845.608632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075845.608649:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075845.608711:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075845.608740:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075846.312581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075846.312707:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075846.632551:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075846.632561:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769075846.632561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075846.632564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075846.632572:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075846.632584:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2832 00000100:00080000:1.0:1769075846.632594:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075846.632600:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075846.632606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075846.632625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075846.632634:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616456704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075846.632667:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075846.632670:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075846.632672:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075846.632676:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2832 00000100:00080000:1.0:1769075846.632680:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075846.632695:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075846.632701:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616456704:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769075846.632703:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075846.632718:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616456832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075846.632726:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075846.632729:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075846.632731:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075846.632735:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2832 00000100:00080000:1.0:1769075846.632738:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075846.632741:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075846.632745:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075846.632755:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616456960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075846.632762:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075846.632768:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1769075846.632768:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075846.632770:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075846.632775:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2832 00000100:00080000:1.0:1769075846.632779:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075846.632782:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075846.632786:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075846.632796:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616457088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075846.632804:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075846.632809:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769075846.632809:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075846/real 1769075846] req@ffff995b868c7100 x1855007616456704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075846.632813:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075846.632820:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2832 00000100:00080000:1.0:1769075846.632826:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075846.632832:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075846.632837:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075846.632843:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616456704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075846.632848:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616457216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075846.632862:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075846.632864:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075846.632868:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769075846.632869:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075846.632874:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075846.632877:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616456704:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075846.632881:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2832 00000100:00080000:1.0:1769075846.632883:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075846.632886:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075846.632893:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075846.632902:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616457344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075846.632911:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075846.632913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456832:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075846.632915:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075846.632919:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075846.632924:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2832 00000100:00080000:1.0:1769075846.632927:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075846.632930:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075846.632934:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075846.632939:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075846.632944:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616457472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075846.632952:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075846/real 1769075846] req@ffff995b868c6300 x1855007616456832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075846.632964:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616456832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075846.632970:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075846.632973:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075846.632976:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075846.632981:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075846.633004:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075846.633012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075846/real 1769075846] req@ffff995b868c6d80 x1855007616456960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075846.633024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616456960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075846.633047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075846.633052:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075846.633056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616456960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075846.633065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075846.633103:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075846.633116:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075846/real 1769075846] req@ffff995b868c5c00 x1855007616457088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075846.633124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616457088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075846.633130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075846.633133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075846.633135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075846.633139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457216:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075846.633152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075846.633159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075846/real 1769075846] req@ffff995b868c4a80 x1855007616457216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075846.633166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616457216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075846.633172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075846.633173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075846.633176:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457216:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075846.633189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616457344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075846.633199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075846.633206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075846/real 1769075846] req@ffff995b868c7800 x1855007616457344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075846.633213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616457344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075846.633219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075846.633220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075846.633222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616457344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075846.633226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616457472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075846.633236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075846.633244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075846/real 1769075846] req@ffff995b868c5500 x1855007616457472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075846.633256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616457472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075846.633265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075846.633268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075846.633272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616457472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075847.336521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075847.336618:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075847.656570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075847.656571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075847.656675:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075847.656676:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075848.360593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075848.360762:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075848.680626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075848.680626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075848.680676:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075848.680700:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075849.384655:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075849.384781:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075849.704613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075849.704617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075849.704657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075849.704679:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075850.408526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075850.408576:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075850.728538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075850.728539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075850.728550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075850.728555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075851.432555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075851.432635:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075851.752604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075851.752621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075851.752641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075851.752685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075851.752710:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075851.752720:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075851.752725:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075851.752738:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2837 00000100:00080000:2.0:1769075851.752753:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075851.752764:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075851.752811:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616457600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075851.752848:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075851.752854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075851.752856:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075851.752862:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2837 00000100:00080000:2.0:1769075851.752866:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075851.752873:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075851.752890:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075851.752915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616457600:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075851.752920:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616457728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075851.752964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075851.752971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075851.752977:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075851.752985:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2837 00000100:00080000:2.0:1769075851.752995:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075851.753007:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769075851.753016:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769075851.753023:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075851.753050:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616457856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075851.753064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075851.753070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075851.753073:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075851.753081:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075851/real 1769075851] req@ffff995b85acbb80 x1855007616457600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075851.753097:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2837 00000100:00080000:2.0:1769075851.753108:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075851.753118:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075851.753126:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075851.753133:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616457600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075851.753149:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616457984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075851.753168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075851.753177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075851.753191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616457600:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075851.753204:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075851.753211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075851.753214:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075851.753220:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2837 00000100:00080000:2.0:1769075851.753234:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075851.753239:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075851.753244:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075851.753257:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616458112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075851.753268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457728:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075851.753276:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075851.753285:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075851.753288:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075851.753294:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2837 00000100:00080000:2.0:1769075851.753300:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075851.753304:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075851.753311:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075851.753325:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616458240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075851.753341:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075851.753345:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075851.753352:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075851.753356:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075851.753363:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2837 00000100:00080000:2.0:1769075851.753369:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075851.753376:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075851/real 1769075851] req@ffff995b85ac8a80 x1855007616457728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075851.753388:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075851.753403:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075851.753419:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616457728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075851.753425:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616458368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075851.753494:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075851.753506:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075851.753526:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075851.753549:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075851.753631:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075851.753657:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075851/real 1769075851] req@ffff995b85aca300 x1855007616457856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075851.753682:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616457856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075851.753695:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075851.753699:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075851.753704:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075851.753712:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457984:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075851.753741:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075851.753754:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075851/real 1769075851] req@ffff995b85acaa00 x1855007616457984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075851.753769:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616457984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075851.753780:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075851.753783:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075851.753787:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616457984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075851.753808:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616458112:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075851.753828:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075851.753839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075851/real 1769075851] req@ffff995b85ac9880 x1855007616458112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075851.753853:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616458112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075851.753872:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075851.753876:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075851.753880:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616458112:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075851.753887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616458240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075851.753904:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075851.753915:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075851/real 1769075851] req@ffff995b85ac9c00 x1855007616458240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075851.753928:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616458240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075851.753939:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075851.753942:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075851.753946:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616458240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075851.753952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616458368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075851.753969:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075851.753979:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075851/real 1769075851] req@ffff995b85acb100 x1855007616458368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075851.753992:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616458368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075851.754002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075851.754005:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075851.754010:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616458368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075852.456583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075852.456719:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075852.776616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075852.776622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075852.776627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075852.776663:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075853.480620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075853.480745:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075853.800604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075853.800612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075853.800617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075853.800689:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075854.504538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075854.504600:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075854.824569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075854.824573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075854.824583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075854.824666:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075855.528549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075855.528623:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075855.848584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075855.848606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075855.848606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075855.848607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075856.552561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075856.552623:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075856.872617:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769075856.872622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075856.872623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075856.872629:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075856.872634:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075856.872643:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2842 00000100:00100000:0.0:1769075856.872644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075856.872654:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075856.872662:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075856.872681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075856.872697:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616458496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075856.872723:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075856.872728:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075856.872729:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075856.872734:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2842 00000100:00080000:2.0:1769075856.872739:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075856.872746:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075856.872755:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075856.872756:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616458496:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075856.872791:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616458624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075856.872802:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075856.872805:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075856.872807:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075856.872810:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2842 00000100:00080000:2.0:1769075856.872813:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075856.872816:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075856.872819:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075856.872830:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616458752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075856.872834:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075856.872843:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075856.872846:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075856.872848:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075856.872851:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2842 00000100:00080000:2.0:1769075856.872855:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075856.872858:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075856.872861:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075856.872870:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616458880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075856.872878:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075856/real 1769075856] req@ffff995b85ac8380 x1855007616458496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075856.872878:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075856.872883:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075856.872887:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075856.872896:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2842 00000100:00080000:2.0:1769075856.872900:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075856.872903:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075856.872907:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075856.872909:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616458496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075856.872918:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616459008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075856.872921:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075856.872930:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769075856.872933:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075856.872937:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616458496:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075856.872938:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075856.872943:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075856.872955:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2842 00000100:00080000:2.0:1769075856.872968:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075856.872974:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075856.872979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616458624:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769075856.872981:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075856.873003:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616459136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075856.873015:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075856.873020:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075856.873022:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075856.873027:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2842 00000800:00000100:0.0:1769075856.873029:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075856.873032:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075856.873037:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075856.873043:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075856.873048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075856/real 1769075856] req@ffff995b85acb800 x1855007616458624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075856.873059:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616459264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075856.873062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616458624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075856.873092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075856.873097:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075856.873100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616458624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075856.873108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616458752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075856.873139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075856.873152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075856/real 1769075856] req@ffff995b85acb100 x1855007616458752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075856.873168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616458752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075856.873190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075856.873193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075856.873195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616458752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075856.873200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616458880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075856.873217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075856.873225:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075856/real 1769075856] req@ffff995b85ac9c00 x1855007616458880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075856.873234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616458880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075856.873240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075856.873243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075856.873245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616458880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075856.873249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459008:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075856.873270:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075856.873277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075856/real 1769075856] req@ffff995b85ac9880 x1855007616459008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075856.873285:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616459008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075856.873291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075856.873293:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075856.873295:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075856.873308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616459136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075856.873320:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075856.873326:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075856/real 1769075856] req@ffff995b85acaa00 x1855007616459136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075856.873334:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616459136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075856.873340:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075856.873342:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075856.873344:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616459136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075856.873347:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616459264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075856.873362:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075856.873374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075856/real 1769075856] req@ffff995b85aca300 x1855007616459264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075856.873396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616459264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075856.873411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075856.873414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075856.873419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616459264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075857.576560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075857.576689:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075857.896648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075857.896686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075857.896693:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075857.896773:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075858.600530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075858.600610:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075858.920566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075858.920571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075858.920577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075858.920599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075859.624573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075859.624691:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075859.944594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075859.944605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075859.944663:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075859.944689:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075860.648578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075860.648722:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075860.968584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075860.968631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075860.968817:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075860.968847:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075861.672582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075861.672701:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075861.992602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075861.992620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075861.992627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075861.992653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075861.992680:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075861.992694:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075861.992701:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075861.992719:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2847 00000100:00080000:1.0:1769075861.992737:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075861.992751:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075861.992805:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616459392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075861.992845:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075861.992851:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075861.992853:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075861.992859:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2847 00000100:00080000:1.0:1769075861.992865:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075861.992872:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075861.992886:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075861.992904:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616459520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075861.992916:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616459392:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075861.992939:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075861.992946:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075861.992949:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075861.992956:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2847 00000100:00080000:1.0:1769075861.992962:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075861.992966:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075861.992973:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075861.992997:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616459648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075861.993019:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075861.993022:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075861.993032:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075861.993039:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075861.993051:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2847 00000100:00080000:1.0:1769075861.993061:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075861.993071:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075861.993081:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075861/real 1769075861] req@ffff995b868c7480 x1855007616459392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769075861.993083:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075861.993126:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616459776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075861.993127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616459392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075861.993184:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075861.993187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075861.993195:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075861.993203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769075861.993204:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075861.993217:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2847 00000100:00080000:1.0:1769075861.993222:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075861.993227:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075861.993231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616459392:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769075861.993233:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075861.993261:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616459904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075861.993279:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075861.993281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459520:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075861.993290:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075861.993297:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075861.993304:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2847 00000100:00080000:1.0:1769075861.993311:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075861.993317:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075861.993325:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769075861.993330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075861.993350:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616460032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075861.993360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075861/real 1769075861] req@ffff995b868c4700 x1855007616459520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075861.993374:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075861.993434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616459520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075861.993435:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075861.993444:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075861.993491:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2847 00000100:00080000:1.0:1769075861.993498:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075861.993507:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769075861.993513:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769075861.993518:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769075861.993527:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075861.993537:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075861.993554:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075861.993559:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616460160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075861.993593:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075861.993617:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075861/real 1769075861] req@ffff995b868c6a00 x1855007616459648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075861.993634:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616459648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075861.993646:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075861.993651:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075861.993655:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075861.993663:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459776:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075861.993682:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075861.993694:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075861/real 1769075861] req@ffff995b868c5500 x1855007616459776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075861.993708:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616459776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075861.993727:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075861.993731:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075861.993735:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075861.993756:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459904:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769075861.993774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075861.993785:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075861/real 1769075861] req@ffff995b868c7800 x1855007616459904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075861.993799:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616459904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075861.993810:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075861.993814:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075861.993819:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616459904:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075861.993830:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616460032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075861.993863:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075861.993884:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075861/real 1769075861] req@ffff995b868c4a80 x1855007616460032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075861.993909:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616460032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075861.993927:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075861.993934:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075861.993941:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616460032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075861.993953:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616460160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769075861.993982:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769075861.993997:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075861/real 1769075861] req@ffff995b868c5c00 x1855007616460160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075861.994011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616460160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075861.994022:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075861.994026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075861.994030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616460160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075862.696575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075862.696718:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075863.016565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075863.016585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075863.016630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075863.016631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075863.720619:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075863.720707:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075864.040540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075864.040540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075864.040542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075864.040576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075864.744611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075864.744711:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075865.064602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075865.064604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075865.064612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075865.064647:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075865.768559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075865.768634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075866.088547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075866.088572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075866.088640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075866.088669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075866.792578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075866.792643:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075867.112562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075867.112574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075867.112585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075867.112596:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075867.112603:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769075867.112608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075867.112608:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075867.112620:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2852 00000100:00080000:3.0:1769075867.112629:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075867.112635:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075867.112665:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616460288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075867.112687:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075867.112690:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075867.112691:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075867.112695:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2852 00000100:00080000:3.0:1769075867.112698:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075867.112701:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075867.112709:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075867.112715:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616460288:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075867.112719:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616460416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075867.112728:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075867.112730:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075867.112731:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075867.112734:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2852 00000100:00080000:3.0:1769075867.112736:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075867.112738:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075867.112741:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075867.112750:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616460544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075867.112756:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075867.112759:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075867.112760:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075867.112762:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2852 00000100:00080000:3.0:1769075867.112765:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075867.112767:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075867.112770:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075867.112777:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616460672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075867.112783:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075867.112786:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075867.112787:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075867.112790:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2852 00000100:00080000:3.0:1769075867.112792:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075867.112794:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075867.112797:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075867.112804:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616460800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075867.112810:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075867.112813:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075867.112814:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075867.112818:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2852 00000100:00080000:3.0:1769075867.112820:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075867.112825:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616460416:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075867.112825:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075867.112831:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075867.112851:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616460928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075867.112861:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075867.112863:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075867.112864:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075867.112870:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2852 00000100:00100000:0.0:1769075867.112871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616460544:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075867.112874:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075867.112880:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075867.112884:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075867.112894:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616461056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075867.112912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616460672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075867.112939:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616460800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769075867.112966:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616460928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769075867.112988:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616461056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075867.816533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075867.816615:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075868.136551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075868.136570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075868.136601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075868.136645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075868.840558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075868.840657:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075869.160539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075869.160541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075869.160548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075869.160599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075869.864557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075869.864734:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075870.184518:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075870.184543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075870.184593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075870.184696:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:0.0:1769075870.184781:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:0.0:1769075870.184814:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769075870.184837:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075870.184875:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075870.184885:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075870.184894:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075870.184903:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075870.184913:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769075870.184925:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769075870.184927:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075867/real 1769075870] req@ffff995b84558700 x1855007616460416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075870.184956:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616460416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075870.184968:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075870.184974:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075870.184978:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616460416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075870.184986:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075867/real 1769075870] req@ffff995b8455a300 x1855007616460544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075870.184997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616460544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075870.185006:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075870.185009:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075870.185013:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616460544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075870.185020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075867/real 1769075870] req@ffff995b8455b100 x1855007616460672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075870.185030:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616460672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075870.185041:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075870.185045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075870.185048:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616460672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075870.185055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075867/real 1769075870] req@ffff995b84559c00 x1855007616460800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075870.185068:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616460800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075870.185077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075870.185081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075870.185084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616460800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075870.185091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075867/real 1769075870] req@ffff995b84559880 x1855007616460288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075870.185103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616460288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075870.185111:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075870.185114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075870.185117:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616460288:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075870.185150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075867/real 1769075870] req@ffff995b8455ad80 x1855007616460928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075870.185161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616460928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075870.185169:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075870.185172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075870.185174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616460928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075870.185180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075867/real 1769075870] req@ffff995b8455b480 x1855007616461056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075870.185190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616461056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075870.185198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075870.185202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075870.185204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616461056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075870.888566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075870.888639:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075871.208531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075871.208532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075871.208566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075871.208633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075871.912580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075871.912673:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075872.232655:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075872.232661:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075872.232677:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075872.232687:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075872.232717:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2857 00000100:00100000:0.0:1769075872.232721:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075872.232739:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075872.232741:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075872.232764:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075872.232792:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075872.232832:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616461184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075872.232865:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075872.232871:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075872.232874:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075872.232883:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2857 00000100:00080000:3.0:1769075872.232888:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075872.232894:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075872.232910:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075872.232926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616461184:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075872.232933:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616461312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075872.232963:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075872.232971:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075872.232975:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075872.232983:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2857 00000100:00080000:3.0:1769075872.232990:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075872.232997:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075872.233008:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075872.233034:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616461440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075872.233053:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075872.233060:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075872.233064:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769075872.233079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075872.233081:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2857 00000100:00080000:3.0:1769075872.233101:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075872.233110:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075872.233121:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075872.233146:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616461568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075872.233170:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075872.233172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075872/real 1769075872] req@ffff995b84558000 x1855007616461184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075872.233182:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075872.233189:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075872.233207:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2857 00000100:00080000:3.0:1769075872.233242:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075872.233254:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075872.233265:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075872.233270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616461184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075872.233297:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616461696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075872.233327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075872.233334:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075872.233344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075872.233347:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075872.233358:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075872.233362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616461184:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075872.233374:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2857 00000100:00080000:3.0:1769075872.233387:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075872.233396:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075872.233408:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075872.233424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616461312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075872.233444:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616461824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075872.233516:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075872.233524:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075872.233529:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075872.233538:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2857 00000100:00080000:3.0:1769075872.233573:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075872.233783:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769075872.233788:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769075872.233802:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075872.233834:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075872/real 1769075872] req@ffff995b8455a680 x1855007616461312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075872.233839:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616461952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075872.233892:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616461312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075872.233910:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075872.233917:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075872.233925:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616461312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075872.233939:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616461440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075872.233987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075872.234012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075872/real 1769075872] req@ffff995b84558a80 x1855007616461440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075872.234036:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616461440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075872.234057:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075872.234064:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075872.234070:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616461440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075872.234083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616461568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075872.234124:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075872.234149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075872/real 1769075872] req@ffff995b84559500 x1855007616461568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075872.234177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616461568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075872.234263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075872.234275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075872.234284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616461568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075872.234297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616461696:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075872.234341:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075872.234366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075872/real 1769075872] req@ffff995b8455aa00 x1855007616461696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075872.234417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616461696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075872.234436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075872.234443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075872.234449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616461696:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075872.234617:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616461824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075872.234670:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075872.234695:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075872/real 1769075872] req@ffff995b8455bb80 x1855007616461824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075872.234722:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616461824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075872.234739:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075872.234746:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075872.234753:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616461824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075872.234766:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616461952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075872.234808:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075872.234833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075872/real 1769075872] req@ffff995b84558000 x1855007616461952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075872.234858:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616461952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075872.234875:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075872.234882:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075872.234889:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616461952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075872.936586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075872.936671:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075873.256620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075873.256622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075873.256664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075873.256725:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075873.960637:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075873.960718:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075874.280589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075874.280595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075874.280620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075874.280673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075874.984642:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075874.984724:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075875.304603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075875.304647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075875.304647:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075875.304708:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075876.008607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075876.008678:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075876.328652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075876.328664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075876.328718:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075876.328788:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075877.032597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075877.032761:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075877.352643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075877.352661:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075877.352662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075877.352665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075877.352703:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075877.352713:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075877.352720:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075877.352739:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2863 00000100:00080000:1.0:1769075877.352756:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075877.352767:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075877.352818:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616462080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075877.352860:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075877.352867:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075877.352872:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075877.352881:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2863 00000100:00080000:1.0:1769075877.352889:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075877.352899:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075877.352910:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616462080:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769075877.352911:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075877.352936:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616462208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075877.352951:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075877.352954:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075877.352957:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075877.352961:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2863 00000100:00080000:1.0:1769075877.352966:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075877.352971:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075877.352979:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075877.352994:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616462336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075877.353005:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075877.353011:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075877.353016:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769075877.353025:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075877.353029:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2863 00000100:00080000:1.0:1769075877.353036:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075877.353040:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075877.353046:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075877.353084:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616462464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075877.353097:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075877.353105:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769075877.353108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075877/real 1769075877] req@ffff995b868c6d80 x1855007616462080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075877.353110:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075877.353126:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2863 00000100:00080000:1.0:1769075877.353166:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075877.353172:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075877.353189:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075877.353192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616462080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075877.353220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769075877.353225:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616462592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075877.353233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075877.353253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616462080:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075877.353254:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075877.353267:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075877.353274:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075877.353282:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2863 00000100:00080000:1.0:1769075877.353288:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075877.353294:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075877.353299:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075877.353319:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616462208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075877.353320:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616462720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075877.353349:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075877.353353:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075877.353356:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075877.353364:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2863 00000100:00080000:1.0:1769075877.353411:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075877.353419:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769075877.353425:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769075877.353436:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075877.353455:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075877/real 1769075877] req@ffff995b868c6300 x1855007616462208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075877.353498:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616462848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075877.353553:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616462208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075877.353563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075877.353567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075877.353571:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616462208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075877.353579:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616462336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075877.353605:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075877.353618:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075877/real 1769075877] req@ffff995b868c7100 x1855007616462336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075877.353630:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616462336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075877.353641:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075877.353644:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075877.353647:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616462336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075877.353653:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616462464:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075877.353670:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075877.353679:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075877/real 1769075877] req@ffff995b868c5c00 x1855007616462464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075877.353710:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616462464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075877.353730:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075877.353734:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075877.353738:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616462464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075877.353745:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616462592:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075877.353764:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075877.353776:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075877/real 1769075877] req@ffff995b868c4a80 x1855007616462592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075877.353788:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616462592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075877.353799:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075877.353802:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075877.353806:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616462592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075877.353829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616462720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075877.353848:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075877.353861:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075877/real 1769075877] req@ffff995b868c7800 x1855007616462720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075877.353875:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616462720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075877.353885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075877.353890:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075877.353897:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616462720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075877.353908:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616462848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075877.353942:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075877.353964:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075877/real 1769075877] req@ffff995b868c5500 x1855007616462848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075877.354004:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616462848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075877.354018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075877.354023:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075877.354028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616462848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075878.056600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075878.056669:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075878.376603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075878.376661:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075878.376669:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075878.376723:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075879.080558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075879.080655:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075879.400602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075879.400609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075879.400648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075879.400742:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075880.104569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075880.104655:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075880.424560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075880.424564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075880.424627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075880.424678:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075881.128608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075881.128700:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075881.448691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075881.448701:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075881.448716:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075881.448719:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075882.152590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075882.152697:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075882.472554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075882.472565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075882.472644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075882.472644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075882.472692:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075882.472700:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075882.472704:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075882.472717:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2868 00000100:00080000:3.0:1769075882.472731:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075882.472738:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075882.472771:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616462976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075882.472805:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075882.472808:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075882.472812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075882.472815:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2868 00000100:00080000:3.0:1769075882.472818:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075882.472824:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075882.472836:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075882.472862:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616463104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075882.472866:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616462976:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075882.472891:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075882.472894:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075882.472896:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075882.472901:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2868 00000100:00080000:3.0:1769075882.472908:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075882.472913:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075882.472919:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075882.472938:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616463232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075882.472952:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075882.472957:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075882.472960:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769075882.472990:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075882.473038:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2868 00000100:00080000:3.0:1769075882.473077:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075882.473093:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075882.473098:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075882.473140:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616463360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075882.473162:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075882.473170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075882/real 1769075882] req@ffff995b8455b480 x1855007616462976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075882.473170:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769075882.473209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616462976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075882.473226:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075882.473232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075882.473240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616462976:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075882.473278:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075882.473289:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2868 00000100:00100000:1.0:1769075882.473297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616463104:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075882.473300:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075882.473360:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075882.473385:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075882.473400:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075882.473412:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075882/real 1769075882] req@ffff995b8455ad80 x1855007616463104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075882.473423:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616463488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075882.473461:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075882.473464:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075882.473466:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075882.473516:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2868 00000100:00100000:1.0:1769075882.473521:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616463104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075882.473529:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075882.473550:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075882.473553:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769075882.473561:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769075882.473568:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075882.473571:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616463104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075882.473598:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616463232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769075882.473611:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616463616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075882.473622:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075882.473625:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075882.473628:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769075882.473636:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075882.473659:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2868 00000100:00080000:3.0:1769075882.473678:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075882.473684:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075882.473690:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075882.473695:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075882/real 1769075882] req@ffff995b84559880 x1855007616463232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075882.473708:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616463744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075882.473732:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616463232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075882.473752:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075882.473758:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075882.473762:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616463232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075882.473771:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616463360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075882.473797:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075882.473817:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075882/real 1769075882] req@ffff995b84559c00 x1855007616463360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075882.473865:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616463360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075882.473882:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075882.473887:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075882.473894:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616463360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075882.473943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616463488:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075882.473984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075882.474000:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075882/real 1769075882] req@ffff995b8455b100 x1855007616463488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075882.474016:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616463488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075882.474028:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075882.474032:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075882.474036:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616463488:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075882.474068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616463616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075882.474097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075882.474112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075882/real 1769075882] req@ffff995b8455a300 x1855007616463616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075882.474127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616463616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075882.474139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075882.474143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075882.474148:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616463616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075882.474155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616463744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075882.474174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075882.474186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075882/real 1769075882] req@ffff995b84558700 x1855007616463744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075882.474199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616463744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075882.474209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075882.474213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075882.474217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616463744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075883.176553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075883.176581:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075883.496562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075883.496581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075883.496608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075883.496673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075884.200659:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075884.200743:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075884.520526:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075884.520545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075884.520608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075884.520642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075885.224505:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075885.224547:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075885.544524:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075885.544525:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075885.544556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075885.544576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075886.248507:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075886.248547:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075886.568516:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075886.568521:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075886.568529:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075886.568559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075887.272593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075887.272686:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075887.592608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075887.592623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075887.592638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075887.592640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769075887.592685:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075887.592693:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075887.592698:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075887.592713:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2873 00000100:00080000:2.0:1769075887.592728:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075887.592737:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075887.592791:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616463872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075887.592820:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075887.592825:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075887.592827:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075887.592833:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2873 00000100:00080000:2.0:1769075887.592837:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075887.592842:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075887.592855:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075887.592878:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616464000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075887.592897:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616463872:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075887.592898:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075887.592924:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075887.592929:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075887.592933:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2873 00000100:00080000:2.0:1769075887.592936:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075887.592941:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075887.592948:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075887.592981:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616464128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075887.592988:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075887.592992:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075887.592995:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075887.592998:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2873 00000100:00080000:2.0:1769075887.593003:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075887.593006:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075887.593013:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075887.593021:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616464256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075887.593028:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075887.593031:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075887.593032:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075887.593035:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2873 00000100:00080000:2.0:1769075887.593037:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075887.593051:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769075887.593054:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769075887.593059:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075887.593075:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616464384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075887.593088:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075887.593092:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075887.593094:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075887.593099:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2873 00000100:00080000:2.0:1769075887.593103:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075887.593106:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075887.593111:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075887.593127:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616464512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075887.593129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075887/real 1769075887] req@ffff995b85ac8a80 x1855007616463872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075887.593171:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075887.593194:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075887.593199:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075887.593208:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2873 00000100:00080000:2.0:1769075887.593212:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075887.593217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616463872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075887.593224:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075887.593238:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075887.593259:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616464640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075887.593263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075887.593288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075887.593295:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616463872:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075887.593338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075887.593367:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075887.593383:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075887/real 1769075887] req@ffff995b85acbb80 x1855007616464000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075887.593400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616464000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075887.593411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075887.593416:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075887.593421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075887.593428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075887.593507:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075887.593535:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075887/real 1769075887] req@ffff995b85aca300 x1855007616464128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075887.593562:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616464128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075887.593582:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075887.593587:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075887.593592:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075887.593601:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075887.593637:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075887.593659:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075887/real 1769075887] req@ffff995b85acaa00 x1855007616464256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075887.593683:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616464256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075887.593702:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075887.593709:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075887.593717:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075887.593730:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464384:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075887.593768:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075887.593785:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075887/real 1769075887] req@ffff995b85ac9880 x1855007616464384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075887.593800:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616464384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075887.593812:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075887.593816:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075887.593820:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464384:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075887.593828:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616464512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075887.593849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075887.593861:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075887/real 1769075887] req@ffff995b85ac9c00 x1855007616464512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075887.593884:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616464512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075887.593894:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075887.593897:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075887.593901:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616464512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075887.593908:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616464640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075887.593926:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075887.593937:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075887/real 1769075887] req@ffff995b85acb100 x1855007616464640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075887.593950:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616464640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075887.593993:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075887.593998:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075887.594003:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616464640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075888.296617:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075888.296711:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075888.616580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075888.616580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075888.616592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075888.616671:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075889.320529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075889.320592:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075889.640579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075889.640621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075889.640647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075889.640685:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075890.344534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075890.344629:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075890.664582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075890.664590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075890.664620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075890.664655:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075891.368541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075891.368637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075891.688561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075891.688569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075891.688599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075891.688632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075892.392575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075892.392663:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075892.712592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075892.712595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075892.712606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769075892.712616:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075892.712626:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075892.712633:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075892.712648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769075892.712648:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2878 00000100:00080000:2.0:1769075892.712672:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075892.712685:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075892.712754:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616464768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075892.712792:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075892.712797:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075892.712798:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075892.712802:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2878 00000100:00080000:2.0:1769075892.712806:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075892.712811:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075892.712821:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075892.712846:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616464896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075892.712856:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075892.712861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616464768:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075892.712861:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075892.712865:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075892.712889:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2878 00000100:00080000:2.0:1769075892.712895:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075892.712903:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075892.712909:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075892.712922:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616465024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075892.712931:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075892.712934:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075892.712936:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075892.712940:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2878 00000100:00080000:2.0:1769075892.712943:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075892.712946:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075892.712951:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075892.712962:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616465152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075892.712976:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075892.712977:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075892.712987:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075892.712990:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075892.712997:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2878 00000100:00080000:2.0:1769075892.713004:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075892.713011:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075892.713020:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075892.713030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075892/real 1769075892] req@ffff995b85acb800 x1855007616464768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075892.713043:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616465280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075892.713060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616464768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075892.713067:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075892.713083:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075892.713084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075892.713087:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075892.713092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769075892.713098:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2878 00000100:00080000:2.0:1769075892.713107:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075892.713114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616464768:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075892.713117:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075892.713129:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075892.713144:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616465408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075892.713151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464896:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075892.713171:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075892.713175:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075892.713178:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075892.713184:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2878 00000100:00080000:2.0:1769075892.713188:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075892.713192:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075892.713197:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075892.713205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075892.713213:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616465536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075892.713230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075892/real 1769075892] req@ffff995b85ac8380 x1855007616464896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075892.713249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616464896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075892.713256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075892.713259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075892.713261:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616464896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075892.713266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075892.713286:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075892.713293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075892/real 1769075892] req@ffff995b85acb100 x1855007616465024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075892.713301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616465024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075892.713308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075892.713311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075892.713313:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075892.713316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465152:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075892.713327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075892.713333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075892/real 1769075892] req@ffff995b85ac9c00 x1855007616465152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075892.713341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616465152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075892.713354:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075892.713357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075892.713359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075892.713363:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465280:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075892.713375:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075892.713382:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075892/real 1769075892] req@ffff995b85ac9880 x1855007616465280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075892.713394:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616465280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075892.713400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075892.713402:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075892.713403:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075892.713416:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616465408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075892.713427:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075892.713433:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075892/real 1769075892] req@ffff995b85acaa00 x1855007616465408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075892.713441:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616465408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075892.713446:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075892.713448:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075892.713450:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616465408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075892.713481:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616465536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075892.713503:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075892.713515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075892/real 1769075892] req@ffff995b85aca300 x1855007616465536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075892.713529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616465536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075892.713541:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075892.713543:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075892.713546:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616465536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075893.416608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075893.416710:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075893.736592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075893.736634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075893.736635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075893.736669:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075894.440584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075894.440683:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075894.760530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075894.760641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075894.760649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075894.760649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075895.464572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075895.464657:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075895.784582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075895.784642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075895.784642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075895.784670:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075896.488580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075896.488654:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075896.808569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075896.808570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075896.808595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075896.808649:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075897.512523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075897.512696:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075897.832575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075897.832576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075897.832582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075897.832625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075897.832679:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075897.832700:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075897.832707:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075897.832723:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2883 00000100:00080000:3.0:1769075897.832742:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075897.832765:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075897.832866:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616465664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075897.832917:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075897.832923:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075897.832926:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075897.832932:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2883 00000100:00080000:3.0:1769075897.832938:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075897.832944:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075897.832957:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075897.832967:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616465664:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075897.832979:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616465792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075897.832994:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075897.832998:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075897.833001:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075897.833006:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2883 00000100:00080000:3.0:1769075897.833011:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075897.833016:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075897.833022:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075897.833035:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616465920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075897.833047:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075897.833055:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075897.833055:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075897.833063:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075897.833071:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2883 00000100:00080000:3.0:1769075897.833077:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075897.833083:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075897.833092:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075897.833113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075897/real 1769075897] req@ffff995b84558700 x1855007616465664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075897.833113:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616466048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075897.833175:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075897.833179:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075897.833181:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075897.833184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616465664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075897.833188:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2883 00000100:00080000:3.0:1769075897.833196:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075897.833202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075897.833202:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769075897.833211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769075897.833212:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075897.833218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616465664:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075897.833229:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616466176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075897.833237:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075897.833240:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075897.833242:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075897.833246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465792:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075897.833250:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2883 00000100:00080000:3.0:1769075897.833254:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075897.833257:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075897.833261:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075897.833272:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616466304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075897.833280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075897.833290:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075897.833293:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075897.833296:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075897.833299:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075897/real 1769075897] req@ffff995b8455a300 x1855007616465792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075897.833303:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2883 00000100:00080000:3.0:1769075897.833310:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075897.833317:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075897.833329:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075897.833336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616465792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075897.833342:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616466432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075897.833349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075897.833359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075897.833365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075897.833371:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075897.833403:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075897.833411:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075897/real 1769075897] req@ffff995b8455b100 x1855007616465920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075897.833424:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616465920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075897.833435:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075897.833439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075897.833442:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616465920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075897.833449:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075897.833471:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075897.833483:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075897/real 1769075897] req@ffff995b84559c00 x1855007616466048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075897.833512:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616466048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075897.833524:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075897.833527:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075897.833529:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075897.833533:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466176:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075897.833548:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075897.833556:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075897/real 1769075897] req@ffff995b84559880 x1855007616466176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075897.833563:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616466176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075897.833570:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075897.833572:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075897.833574:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466176:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075897.833586:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616466304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075897.833597:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075897.833603:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075897/real 1769075897] req@ffff995b8455ad80 x1855007616466304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075897.833611:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616466304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075897.833617:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075897.833618:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075897.833621:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616466304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075897.833625:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616466432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075897.833653:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075897.833660:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075897/real 1769075897] req@ffff995b8455b480 x1855007616466432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075897.833668:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616466432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075897.833675:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075897.833677:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075897.833679:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616466432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075898.536530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075898.536618:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075898.856601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075898.856609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075898.856610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075898.856688:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075899.560642:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075899.560749:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075899.880580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075899.880580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075899.880647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075899.880652:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075900.584532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075900.584593:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075900.904584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075900.904615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075900.904626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075900.904647:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075901.608545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075901.608593:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075901.928584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075901.928585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075901.928597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075901.928657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075902.632523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075902.632560:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075902.952567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075902.952568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075902.952571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075902.952590:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075902.952599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075902.952602:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075902.952604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075902.952611:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2888 00000100:00080000:0.0:1769075902.952622:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075902.952640:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075902.952683:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83505180 x1855007616466560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075902.952730:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075902.952734:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075902.952735:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075902.952738:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2888 00000100:00080000:0.0:1769075902.952752:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075902.952756:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075902.952763:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075902.952774:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504000 x1855007616466688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075902.952784:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075902.952786:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075902.952788:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075902.952793:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2888 00000100:00100000:1.0:1769075902.952793:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616466560:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769075902.952799:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075902.952803:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075902.952806:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075902.952813:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507100 x1855007616466816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075902.952819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075902.952822:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075902.952823:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075902.952826:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2888 00000100:00080000:0.0:1769075902.952828:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075902.952830:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075902.952835:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075902.952841:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83506300 x1855007616466944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075902.952847:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075902.952850:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075902.952851:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075902.952853:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2888 00000100:00080000:0.0:1769075902.952855:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075902.952869:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769075902.952870:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769075902.952873:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075902.952882:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83507800 x1855007616467072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075902.952889:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075902.952891:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075902.952892:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075902.952896:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2888 00000100:00080000:0.0:1769075902.952898:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075902.952901:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075902.952904:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075902.952908:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075902/real 1769075902] req@ffff995b83505180 x1855007616466560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075902.952915:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b83504e00 x1855007616467200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075902.952924:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075902.952927:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769075902.952930:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83505180 x1855007616466560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075902.952931:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075902.952938:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2888 00000100:00080000:0.0:1769075902.952943:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075902.952947:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769075902.952949:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769075902.952950:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075902.952956:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075902.952963:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616467328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075902.952964:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83505180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616466560:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075902.953001:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075902.953017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075902.953026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075902/real 1769075902] req@ffff995b83504000 x1855007616466688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.953035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504000 x1855007616466688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.953041:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.953044:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075902.953046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075902.953051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075902.953062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075902.953069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075902/real 1769075902] req@ffff995b83507100 x1855007616466816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.953077:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507100 x1855007616466816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.953085:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.953088:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075902.953090:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075902.953094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466944:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075902.953104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075902.953109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075902/real 1769075902] req@ffff995b83506300 x1855007616466944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.953116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83506300 x1855007616466944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.953122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.953123:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075902.953125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83506300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616466944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075902.953128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467072:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075902.953137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075902.953142:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075902/real 1769075902] req@ffff995b83507800 x1855007616467072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.953149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83507800 x1855007616467072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.953155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.953157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075902.953159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83507800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075902.953162:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616467200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075902.953172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075902.953177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075902/real 1769075902] req@ffff995b83504e00 x1855007616467200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.953185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b83504e00 x1855007616467200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.953190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.953192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075902.953194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b83504e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616467200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075902.953197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616467328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075902.953205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075902.953211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075902/real 1769075902] req@ffff995b84640a80 x1855007616467328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.953218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616467328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.953223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.953225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075902.953226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616467328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075903.656525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075903.656574:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075903.976568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075903.976569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075903.976589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075903.976620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075904.680506:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075904.680611:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075905.000580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075905.000580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075905.000584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075905.000605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075905.704642:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075905.704837:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075906.024571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075906.024595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075906.024669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075906.024701:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075906.728541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075906.728664:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075907.048603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075907.048603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075907.048677:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075907.048743:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075907.752680:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075907.752760:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075908.072534:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075908.072534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075908.072549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075908.072555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075908.072560:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075908.072569:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2893 00000100:00100000:1.0:1769075908.072579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075908.072580:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075908.072590:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075908.072596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075908.072638:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616467456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075908.072663:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075908.072666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075908.072668:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075908.072670:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2893 00000100:00080000:3.0:1769075908.072673:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075908.072678:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075908.072686:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075908.072695:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616467456:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769075908.072698:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616467584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075908.072708:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075908.072710:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075908.072712:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075908.072714:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2893 00000100:00080000:3.0:1769075908.072716:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075908.072719:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075908.072723:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075908.072732:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616467712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075908.072738:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075908.072740:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075908.072741:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075908.072744:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2893 00000100:00080000:3.0:1769075908.072747:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075908.072750:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075908.072753:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769075908.072760:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769075908.072764:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616467840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075908.072772:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075908.072775:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075908.072776:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075908.072780:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2893 00000100:00080000:3.0:1769075908.072782:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075908.072784:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075908.072787:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075908.072795:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616467968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075908.072801:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075908/real 1769075908] req@ffff995b84558000 x1855007616467456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075908.072803:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075908.072812:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075908.072815:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075908.072822:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2893 00000100:00080000:3.0:1769075908.072826:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075908.072829:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075908.072832:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616467456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075908.072834:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075908.072846:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769075908.072848:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616468096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075908.072854:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075908.072861:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616467456:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075908.072874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075908.072877:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075908.072879:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075908.072882:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2893 00000100:00080000:3.0:1769075908.072884:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075908.072888:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467584:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075908.072889:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075908.072897:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075908.072908:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616468224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075908.072916:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075908.072930:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075908/real 1769075908] req@ffff995b8455bb80 x1855007616467584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075908.072939:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616467584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075908.072945:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075908.072947:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075908.072950:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075908.072954:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075908.072964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075908.072969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075908/real 1769075908] req@ffff995b8455aa00 x1855007616467712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075908.072982:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616467712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075908.072988:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075908.072990:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075908.072991:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075908.072995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467840:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075908.073005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075908.073010:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075908/real 1769075908] req@ffff995b84559500 x1855007616467840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075908.073018:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616467840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075908.073024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075908.073026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075908.073028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467840:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075908.073031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467968:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075908.073040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075908.073055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075908/real 1769075908] req@ffff995b84558a80 x1855007616467968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075908.073062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616467968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075908.073073:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075908.073075:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075908.073077:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616467968:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075908.073080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616468096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075908.073090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075908.073100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075908/real 1769075908] req@ffff995b8455a680 x1855007616468096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075908.073121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616468096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075908.073130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075908.073133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075908.073137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616468096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075908.073156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616468224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075908.073185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075908.073194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075908/real 1769075908] req@ffff995b84558000 x1855007616468224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075908.073203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616468224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075908.073208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075908.073210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075908.073213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616468224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075908.776570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075908.776643:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075909.096564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075909.096567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075909.096571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075909.096617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075909.800571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075909.800665:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075910.120538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075910.120556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075910.120571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075910.120622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075910.824529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075910.824626:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075911.144578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075911.144585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075911.144595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075911.144639:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075911.848540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075911.848627:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075912.168525:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075912.168542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075912.168573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075912.168604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075912.872571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075912.872654:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075913.192578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075913.192609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075913.192652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769075913.192687:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075913.192693:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075913.192697:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075913.192705:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075913.192711:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2898 00000100:00080000:2.0:1769075913.192729:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075913.192741:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075913.192782:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616468352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075913.192806:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075913.192809:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075913.192810:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075913.192813:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2898 00000100:00080000:2.0:1769075913.192816:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075913.192818:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075913.192825:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075913.192834:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616468480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075913.192844:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075913.192848:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075913.192850:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075913.192853:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2898 00000100:00080000:2.0:1769075913.192855:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075913.192857:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075913.192860:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075913.192882:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616468608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075913.192891:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075913.192899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616468352:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075913.192900:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075913.192909:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075913.192926:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2898 00000100:00080000:2.0:1769075913.192938:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075913.192941:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075913.192945:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075913.192953:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616468736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075913.192959:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075913.192961:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075913.192963:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075913.192966:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2898 00000100:00080000:2.0:1769075913.192968:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075913.192972:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075913.192975:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075913.192984:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616468864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075913.192992:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075913.192994:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075913.192996:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075913.192999:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2898 00000100:00080000:2.0:1769075913.193002:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075913.193005:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075913.193011:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075913.193018:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616468992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075913.193025:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075913.193027:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075913.193028:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075913.193031:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2898 00000100:00080000:2.0:1769075913.193034:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075913.193036:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075913.193041:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075913.193049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075913.193058:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616469120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075913.193120:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075913/real 1769075913] req@ffff995b85acbb80 x1855007616468352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075913.193147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616468352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075913.193163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075913.193169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075913.193174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616468352:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075913.193184:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616468480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075913.193211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075913.193226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075913/real 1769075913] req@ffff995b85ac8a80 x1855007616468480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075913.193241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616468480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075913.193272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075913.193276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075913.193280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616468480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075913.193309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616468608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075913.193343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075913.193356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075913/real 1769075913] req@ffff995b85aca300 x1855007616468608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075913.193371:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616468608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075913.193384:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075913.193388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075913.193392:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616468608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075913.193400:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616468736:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075913.193420:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075913.193431:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075913/real 1769075913] req@ffff995b85acaa00 x1855007616468736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075913.193446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616468736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075913.193457:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075913.193461:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075913.193465:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616468736:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075913.193471:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616468864:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075913.193488:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075913.193499:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075913/real 1769075913] req@ffff995b85ac9880 x1855007616468864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075913.193512:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616468864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075913.193595:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075913.193600:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075913.193603:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616468864:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075913.193610:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616468992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075913.193634:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075913.193646:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075913/real 1769075913] req@ffff995b85ac9c00 x1855007616468992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075913.193679:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616468992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075913.193690:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075913.193694:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075913.193698:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616468992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075913.193705:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616469120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075913.193726:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075913.193739:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075913/real 1769075913] req@ffff995b85acb100 x1855007616469120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075913.193753:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616469120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075913.193764:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075913.193767:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075913.193771:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616469120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075913.896610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075913.896689:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075914.216585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075914.216591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075914.216617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075914.216626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075914.920563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075914.920622:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075915.240590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075915.240590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075915.240600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075915.240632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075915.944554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075915.944617:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075916.264627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075916.264651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075916.264690:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075916.264699:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075916.968516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075916.968609:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075917.288556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075917.288572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075917.288572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075917.288576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075917.992550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075917.992612:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075918.312557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075918.312564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075918.312565:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075918.312574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075918.312578:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075918.312588:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2904 00000100:00100000:2.0:1769075918.312592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769075918.312597:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075918.312606:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075918.312636:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616469248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075918.312638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075918.312661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075918.312664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075918.312666:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075918.312670:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2904 00000100:00080000:0.0:1769075918.312673:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075918.312676:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075918.312684:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075918.312693:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616469376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075918.312700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075918.312702:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075918.312704:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075918.312707:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2904 00000100:00080000:0.0:1769075918.312709:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075918.312712:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075918.312716:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075918.312717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616469248:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075918.312726:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616469504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075918.312735:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075918.312738:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075918.312740:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075918.312744:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2904 00000100:00080000:0.0:1769075918.312748:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075918.312759:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075918.312764:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075918.312775:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616469632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075918.312781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769075918.312787:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075918.312790:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075918.312792:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075918.312795:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2904 00000100:00080000:0.0:1769075918.312799:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075918.312802:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075918.312805:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075918.312812:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616469760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075918.312819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075918.312821:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075918/real 1769075918] req@ffff995b84641f80 x1855007616469248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075918.312822:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075918.312826:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075918.312834:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2904 00000100:00080000:0.0:1769075918.312838:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075918.312841:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075918.312844:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075918.312846:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616469248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075918.312857:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616469888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075918.312857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075918.312866:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075918.312873:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616469248:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769075918.312875:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075918.312883:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769075918.312885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616469376:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769075918.312887:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075918.312898:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2904 00000100:00080000:0.0:1769075918.312905:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075918.312910:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769075918.312912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769075918.312922:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075918.312927:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075918/real 1769075918] req@ffff995b84640380 x1855007616469376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075918.312939:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616470016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075918.312940:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616469376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075918.312969:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075918.312975:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075918.312978:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616469376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075918.313000:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616469504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075918.313015:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075918.313022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075918/real 1769075918] req@ffff995b84640700 x1855007616469504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075918.313030:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616469504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075918.313038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075918.313040:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075918.313043:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616469504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075918.313050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616469632:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075918.313064:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075918.313077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075918/real 1769075918] req@ffff995b84642680 x1855007616469632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075918.313091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616469632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075918.313100:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075918.313104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075918.313109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616469632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075918.313117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616469760:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075918.313138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075918.313152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075918/real 1769075918] req@ffff995b84641c00 x1855007616469760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075918.313167:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616469760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075918.313177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075918.313181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075918.313184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616469760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075918.313190:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616469888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075918.313210:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075918.313218:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075918/real 1769075918] req@ffff995b84640000 x1855007616469888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075918.313237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616469888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075918.313247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075918.313249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075918.313251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616469888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075918.313256:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616470016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075918.313274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075918.313285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075918/real 1769075918] req@ffff995b84641180 x1855007616470016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075918.313299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616470016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075918.313308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075918.313311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075918.313314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616470016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075919.016566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075919.016619:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075919.336550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075919.336563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075919.336564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075919.336596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075920.040552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075920.040677:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075920.360565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075920.360566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075920.360566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075920.360593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075921.064565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075921.064631:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075921.384566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075921.384566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075921.384566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075921.384596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075922.088534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075922.088591:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075922.408577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075922.408606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075922.408610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075922.408640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075923.112561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075923.112647:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075923.432555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075923.432562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075923.432565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075923.432590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075923.432643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075923.432650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075923.432654:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075923.432662:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2909 00000100:00080000:2.0:1769075923.432673:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075923.432680:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075923.432726:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616470144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075923.432753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075923.432759:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075923.432761:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075923.432766:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2909 00000100:00080000:2.0:1769075923.432770:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075923.432776:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075923.432785:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075923.432792:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616470144:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075923.432799:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616470272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075923.432809:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075923.432812:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075923.432814:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075923.432817:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2909 00000100:00080000:2.0:1769075923.432820:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075923.432823:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075923.432827:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075923.432837:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616470400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075923.432845:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075923.432848:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075923.432849:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075923.432853:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2909 00000800:00000100:1.0:1769075923.432855:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075923.432856:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075923.432859:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075923.432865:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075923.432873:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616470528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075923.432882:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075923.432885:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075923.432886:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075923.432890:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2909 00000100:00100000:1.0:1769075923.432892:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075923/real 1769075923] req@ffff995b85ac8380 x1855007616470144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075923.432893:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075923.432896:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075923.432901:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075923.432910:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616470144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075923.432910:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616470656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075923.432920:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075923.432922:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075923.432924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075923.432925:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075923.432927:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769075923.432929:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2909 00000100:00100000:1.0:1769075923.432931:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616470144:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075923.432932:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075923.432936:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075923.432939:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075923.432947:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616470784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075923.432952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616470272:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075923.432957:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075923.432960:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075923.432963:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075923.432966:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2909 00000100:00080000:2.0:1769075923.432969:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075923.432972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075923.432974:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075923.432978:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075923.432984:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075923/real 1769075923] req@ffff995b85acb800 x1855007616470272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075923.432987:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616470912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075923.432998:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616470272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075923.433006:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075923.433010:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075923.433013:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616470272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075923.433019:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616470400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075923.433049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075923.433057:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075923/real 1769075923] req@ffff995b85acb100 x1855007616470400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075923.433070:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616470400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075923.433077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075923.433080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075923.433082:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616470400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075923.433086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616470528:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075923.433097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075923.433103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075923/real 1769075923] req@ffff995b85ac9c00 x1855007616470528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075923.433110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616470528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075923.433116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075923.433118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075923.433119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616470528:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075923.433123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616470656:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075923.433132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075923.433138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075923/real 1769075923] req@ffff995b85ac9880 x1855007616470656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075923.433145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616470656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075923.433155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075923.433157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075923.433159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616470656:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075923.433170:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616470784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075923.433180:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075923.433185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075923/real 1769075923] req@ffff995b85acaa00 x1855007616470784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075923.433192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616470784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075923.433198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075923.433199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075923.433201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616470784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075923.433204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616470912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075923.433213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075923.433219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075923/real 1769075923] req@ffff995b85aca300 x1855007616470912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075923.433226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616470912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075923.433231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075923.433232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075923.433235:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616470912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075924.136543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075924.136619:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075924.456577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075924.456577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075924.456591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075924.456647:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075925.160553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075925.160613:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075925.480566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075925.480585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075925.480592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075925.480664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075926.184524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075926.184677:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075926.504624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075926.504625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075926.504662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075926.504742:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075927.208541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075927.208628:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075927.528578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075927.528579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075927.528630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075927.528633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075928.232531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075928.232611:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075928.552545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075928.552549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769075928.552549:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075928.552557:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075928.552562:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075928.552588:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2914 00000100:00100000:1.0:1769075928.552595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075928.552598:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769075928.552604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075928.552606:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075928.552640:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616471040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075928.552660:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075928.552665:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075928.552666:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075928.552670:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2914 00000100:00080000:2.0:1769075928.552673:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075928.552676:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075928.552684:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075928.552691:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616471040:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075928.552695:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616471168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075928.552704:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075928.552706:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075928.552708:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075928.552710:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2914 00000100:00080000:2.0:1769075928.552713:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075928.552715:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075928.552718:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075928.552738:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616471296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075928.552744:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075928.552752:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075928.552753:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075928.552756:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2914 00000100:00080000:2.0:1769075928.552759:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075928.552761:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075928.552767:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769075928.552768:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075928.552777:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616471424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075928.552786:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075928.552788:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075928.552789:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075928.552791:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2914 00000100:00080000:2.0:1769075928.552794:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075928.552796:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075928.552800:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075928.552808:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616471552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075928.552820:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075928.552823:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075928/real 1769075928] req@ffff995b85ac8a80 x1855007616471040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075928.552823:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075928.552826:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075928.552834:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2914 00000100:00080000:2.0:1769075928.552839:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075928.552843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075928.552846:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075928.552852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616471040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075928.552855:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616471680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075928.552878:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075928.552878:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075928.552884:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075928.552887:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769075928.552888:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075928.552894:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616471040:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075928.552895:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2914 00000100:00080000:2.0:1769075928.552903:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075928.552905:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075928.552908:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075928.552916:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616471808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075928.552921:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616471168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075928.552952:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075928.552963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075928/real 1769075928] req@ffff995b85acbb80 x1855007616471168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075928.552971:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616471168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075928.552977:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075928.552980:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075928.552982:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616471168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075928.552986:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616471296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075928.552996:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075928.553002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075928/real 1769075928] req@ffff995b85aca300 x1855007616471296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075928.553009:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616471296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075928.553016:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075928.553018:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075928.553020:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616471296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075928.553024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616471424:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075928.553034:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075928.553040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075928/real 1769075928] req@ffff995b85acaa00 x1855007616471424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075928.553047:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616471424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075928.553053:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075928.553054:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075928.553057:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616471424:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075928.553061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616471552:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075928.553070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075928.553076:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075928/real 1769075928] req@ffff995b85ac9880 x1855007616471552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075928.553083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616471552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075928.553089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075928.553091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075928.553092:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616471552:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075928.553096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616471680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075928.553121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075928.553132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075928/real 1769075928] req@ffff995b85ac9c00 x1855007616471680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075928.553144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616471680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075928.553152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075928.553156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075928.553159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616471680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075928.553188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616471808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075928.553206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075928.553215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075928/real 1769075928] req@ffff995b85acb100 x1855007616471808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075928.553223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616471808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075928.553229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075928.553231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075928.553235:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616471808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075929.256508:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075929.256584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075929.576539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075929.576556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075929.576563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075929.576584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075930.280565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075930.280635:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075930.600550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075930.600558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075930.600589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075930.600681:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075931.304569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075931.304627:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075931.624532:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075931.624543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075931.624562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075931.624574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075932.328530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075932.328581:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075932.648563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075932.648563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075932.648563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075932.648601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075933.352547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075933.352600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075933.672551:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075933.672554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769075933.672562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769075933.672564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075933.672566:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075933.672577:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2919 00000100:00100000:0.0:1769075933.672585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075933.672588:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075933.672597:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075933.672614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075933.672632:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616471936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075933.672665:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075933.672668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075933.672669:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075933.672673:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2919 00000100:00080000:2.0:1769075933.672676:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075933.672678:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075933.672686:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075933.672694:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616472064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075933.672702:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616471936:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075933.672704:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075933.672711:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075933.672723:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075933.672727:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2919 00000100:00080000:2.0:1769075933.672729:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075933.672733:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075933.672739:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075933.672753:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616472192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075933.672763:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075933.672767:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075933.672769:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075933.672773:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2919 00000100:00080000:2.0:1769075933.672778:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075933.672783:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075933.672790:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075933.672802:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616472320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075933.672812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075933.672815:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075933.672816:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075933.672819:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2919 00000100:00100000:1.0:1769075933.672824:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472064:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075933.672825:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075933.672833:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075933.672839:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075933.672862:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616472448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075933.672872:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075933.672876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769075933.672880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472192:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075933.672881:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075933.672891:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2919 00000100:00080000:2.0:1769075933.672897:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075933.672903:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075933.672907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472320:192.168.204.132@tcp:8:kworker.0 00000008:00100000:2.0:1769075933.672924:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075933.672939:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616472576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075933.672943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472448:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769075933.672952:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075933.672956:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075933.672958:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075933.672963:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2919 00000100:00080000:2.0:1769075933.672968:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075933.672971:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616472576:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075933.672975:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075933.672992:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075933.673003:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616472704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075933.673040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616472704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075934.376541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075934.376588:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075934.696571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075934.696573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075934.696587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075934.696617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075935.400534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075935.400593:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075935.720562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075935.720573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075935.720580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075935.720625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075936.424534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075936.424620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075936.744533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075936.744538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075936.744560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075936.744658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769075936.744760:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769075936.744806:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769075936.744824:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075936.744877:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075936.744888:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075936.744899:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075936.744922:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769075936.744935:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769075936.744944:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075933/real 1769075936] req@ffff995b85ac8380 x1855007616472064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1769075936.744946:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769075936.744975:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616472064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075936.744988:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075936.744994:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075936.744999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075936.745009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075933/real 1769075936] req@ffff995b85acb100 x1855007616472192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075936.745023:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616472192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075936.745036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075936.745041:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075936.745047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075936.745055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075933/real 1769075936] req@ffff995b85ac9c00 x1855007616472320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075936.745066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616472320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075936.745074:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075936.745078:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075936.745089:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472320:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075936.745096:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075933/real 1769075936] req@ffff995b85acb800 x1855007616471936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075936.745111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616471936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075936.745119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075936.745128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075936.745166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616471936:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075936.745197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075933/real 1769075936] req@ffff995b85aca300 x1855007616472704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075936.745211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616472704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075936.745220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075936.745223:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075936.745228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616472704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075936.745235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075933/real 1769075936] req@ffff995b85acaa00 x1855007616472576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075936.745246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616472576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075936.745254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075936.745257:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075936.745260:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616472576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075936.745267:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075933/real 1769075936] req@ffff995b85ac9880 x1855007616472448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075936.745277:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616472448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075936.745286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075936.745289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075936.745293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472448:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075937.448526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075937.448581:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075937.768579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075937.768579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075937.768579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075937.768718:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075938.472523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075938.472589:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075938.792524:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769075938.792539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075938.792542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769075938.792546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075938.792551:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075938.792559:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2924 00000100:00100000:1.0:1769075938.792567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769075938.792569:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075938.792578:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075938.792580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075938.792609:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616472832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075938.792630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075938.792633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075938.792634:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075938.792638:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2924 00000100:00080000:0.0:1769075938.792640:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075938.792644:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075938.792652:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075938.792666:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616472832:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075938.792667:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616472960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075938.792684:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075938.792688:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075938.792690:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075938.792694:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2924 00000100:00080000:0.0:1769075938.792699:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075938.792703:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075938.792710:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075938.792721:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616473088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075938.792737:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075938.792741:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075938.792743:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075938.792748:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2924 00000100:00080000:0.0:1769075938.792753:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075938.792758:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769075938.792774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769075938.792775:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075938.792791:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616473216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075938.792800:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075938.792804:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075938.792806:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075938.792810:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2924 00000100:00080000:0.0:1769075938.792815:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075938.792821:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075938/real 1769075938] req@ffff995b84640e00 x1855007616472832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075938.792829:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075938.792838:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075938.792846:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616472832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075938.792852:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616473344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075938.792868:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075938.792871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075938.792876:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075938.792881:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616472832:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769075938.792893:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075938.792896:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075938.792901:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2924 00000100:00080000:0.0:1769075938.792907:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075938.792912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472960:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769075938.792913:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075938.792923:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075938.792938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616473472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075938.792959:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075938.792961:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769075938.792965:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075938.792968:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075938.792974:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2924 00000100:00080000:0.0:1769075938.792979:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075938.792985:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075938.792986:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075938/real 1769075938] req@ffff995b84642a00 x1855007616472960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769075938.792992:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075938.793008:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616473600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075938.793022:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616472960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075938.793038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075938.793044:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075938.793050:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616472960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075938.793060:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075938.793082:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075938.793094:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075938/real 1769075938] req@ffff995b84643800 x1855007616473088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075938.793145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616473088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075938.793163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075938.793167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075938.793171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075938.793178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473216:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075938.793203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075938.793215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075938/real 1769075938] req@ffff995b84643b80 x1855007616473216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075938.793229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616473216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075938.793239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075938.793243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075938.793246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473216:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075938.793252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473344:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075938.793278:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075938.793291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075938/real 1769075938] req@ffff995b84641880 x1855007616473344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075938.793305:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616473344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075938.793316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075938.793320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075938.793325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473344:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075938.793353:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616473472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075938.793376:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075938.793388:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075938/real 1769075938] req@ffff995b84641500 x1855007616473472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075938.793400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616473472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075938.793408:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075938.793411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075938.793414:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616473472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075938.793419:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616473600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075938.793438:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075938.793449:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075938/real 1769075938] req@ffff995b84643480 x1855007616473600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075938.793491:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616473600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075938.793500:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075938.793504:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075938.793508:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616473600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075939.496533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075939.496627:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075939.816583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075939.816602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075939.816612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075939.816651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075940.520559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075940.520679:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075940.840547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075940.840547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075940.840561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075940.840654:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075941.544558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075941.544628:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075941.864568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075941.864569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075941.864589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075941.864671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075942.568536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075942.568622:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075942.888547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075942.888553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075942.888559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075942.888562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075943.592523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075943.592617:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075943.912549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075943.912550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075943.912558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075943.912576:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075943.912582:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075943.912586:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075943.912594:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2929 00000100:00100000:2.0:1769075943.912599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769075943.912604:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075943.912613:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769075943.912647:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616473728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075943.912669:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075943.912672:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075943.912673:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075943.912676:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2929 00000100:00080000:0.0:1769075943.912679:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075943.912682:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075943.912690:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075943.912700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616473728:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769075943.912701:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616473856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075943.912713:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075943.912715:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075943.912716:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075943.912719:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2929 00000100:00080000:0.0:1769075943.912722:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075943.912724:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075943.912727:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075943.912735:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616473984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075943.912742:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075943.912744:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075943.912746:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075943.912748:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2929 00000100:00080000:0.0:1769075943.912758:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075943.912761:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769075943.912766:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769075943.912768:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769075943.912778:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616474112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075943.912785:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075943.912787:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075943.912788:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075943.912791:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2929 00000100:00080000:0.0:1769075943.912793:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075943.912796:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075943.912799:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075943.912804:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075943/real 1769075943] req@ffff995b84642d80 x1855007616473728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075943.912809:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616474240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075943.912829:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075943.912832:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075943.912834:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075943.912837:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616473728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075943.912840:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2929 00000100:00080000:0.0:1769075943.912845:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075943.912852:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769075943.912855:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769075943.912857:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769075943.912863:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769075943.912867:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616474368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075943.912870:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616473728:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769075943.912887:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075943.912889:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075943.912891:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075943.912895:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2929 00000100:00080000:0.0:1769075943.912898:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075943.912898:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473856:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769075943.912903:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769075943.912908:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075943.912917:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616474496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075943.912925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075943.912938:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075943/real 1769075943] req@ffff995b84643100 x1855007616473856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075943.912948:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616473856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075943.912954:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075943.912958:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075943.912961:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075943.912966:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075943.912976:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075943.912982:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075943/real 1769075943] req@ffff995b84643480 x1855007616473984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075943.912989:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616473984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075943.912996:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075943.912998:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075943.912999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616473984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075943.913003:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616474112:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075943.913042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075943.913050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075943/real 1769075943] req@ffff995b84641500 x1855007616474112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075943.913058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616474112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075943.913064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075943.913066:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075943.913068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616474112:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075943.913072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616474240:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075943.913088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075943.913098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075943/real 1769075943] req@ffff995b84641880 x1855007616474240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075943.913110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616474240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075943.913120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075943.913124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075943.913128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616474240:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075943.913134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616474368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075943.913153:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075943.913161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075943/real 1769075943] req@ffff995b84643b80 x1855007616474368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075943.913168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616474368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075943.913174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075943.913176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075943.913178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616474368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075943.913192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616474496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075943.913203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075943.913209:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075943/real 1769075943] req@ffff995b84643800 x1855007616474496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075943.913216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616474496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075943.913222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075943.913224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075943.913226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616474496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075944.616543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075944.616616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075944.936574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075944.936574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075944.936586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075944.936610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075945.640504:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075945.640659:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075945.960596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075945.960596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075945.960605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075945.960644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075946.664540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075946.664597:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075946.984527:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075946.984545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075946.984546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075946.984573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075947.688548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075947.688609:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075948.008515:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075948.008532:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075948.008537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075948.008565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075948.712553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075948.712639:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075949.032540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075949.032603:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075949.032613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075949.032614:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075949.032621:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075949.032624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075949.032637:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2934 00000100:00080000:1.0:1769075949.032652:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075949.032661:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075949.032709:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616474624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075949.032745:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075949.032748:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075949.032750:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075949.032753:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2934 00000100:00080000:1.0:1769075949.032757:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075949.032760:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075949.032771:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075949.032795:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616474752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075949.032804:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075949.032809:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616474624:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075949.032811:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075949.032818:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075949.032828:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2934 00000100:00080000:1.0:1769075949.032831:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075949.032834:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075949.032838:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075949.032851:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616474880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075949.032858:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075949.032863:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075949.032864:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075949.032869:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2934 00000100:00080000:1.0:1769075949.032878:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075949.032880:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075949.032890:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075949.032906:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616475008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075949.032914:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075949.032921:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1769075949.032922:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075949.032924:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075949.032957:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2934 00000100:00080000:1.0:1769075949.032963:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075949.032967:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075949.032972:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075949.032986:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616475136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075949.032994:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075949.032997:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075949.032999:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075949.033002:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2934 00000100:00080000:1.0:1769075949.033005:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075949.033008:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075949.033023:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075949.033035:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616475264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075949.033041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075949/real 1769075949] req@ffff995b868c6a00 x1855007616474624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075949.033050:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075949.033062:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075949.033066:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075949.033072:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616474624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075949.033073:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2934 00000100:00080000:1.0:1769075949.033084:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075949.033106:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075949.033111:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075949.033135:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616475392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075949.033135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075949.033184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075949.033194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616474624:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075949.033258:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616474752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075949.033309:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075949.033327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075949/real 1769075949] req@ffff995b868c4700 x1855007616474752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075949.033344:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616474752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075949.033355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075949.033359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075949.033363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616474752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075949.033372:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616474880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075949.033396:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075949.033409:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075949/real 1769075949] req@ffff995b868c7480 x1855007616474880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075949.033423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616474880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075949.033435:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075949.033439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075949.033444:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616474880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075949.033452:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075949.033526:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000800:00000100:2.0:1769075949.033545:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075949.033575:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075949/real 1769075949] req@ffff995b868c5500 x1855007616475008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075949.033595:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616475008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075949.033610:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075949.033615:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075949.033620:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075949.033630:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475136:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075949.033657:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075949.033673:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075949/real 1769075949] req@ffff995b868c7800 x1855007616475136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075949.033685:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616475136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075949.033694:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075949.033697:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075949.033700:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475136:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075949.033706:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616475264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075949.033738:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075949.033746:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075949/real 1769075949] req@ffff995b868c4a80 x1855007616475264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075949.033754:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616475264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075949.033765:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075949.033768:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075949.033770:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616475264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075949.033774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616475392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075949.033811:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075949.033827:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075949/real 1769075949] req@ffff995b868c5c00 x1855007616475392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075949.033845:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616475392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075949.033859:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075949.033864:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075949.033870:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616475392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075949.736538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075949.736758:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075950.056591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075950.056608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075950.056651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075950.056663:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075950.760529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075950.760644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075951.080513:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075951.080546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075951.080546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075951.080560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075951.784528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075951.784578:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075952.104548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075952.104559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075952.104572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075952.104591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075952.808506:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075952.808579:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075953.128525:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075953.128546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075953.128550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075953.128554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075953.832536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075953.832597:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075954.152517:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075954.152540:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075954.152565:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769075954.152567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075954.152572:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075954.152589:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2939 00000100:00080000:1.0:1769075954.152606:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075954.152630:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075954.152664:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616475520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075954.152713:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075954.152719:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075954.152722:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075954.152728:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2939 00000100:00080000:1.0:1769075954.152736:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075954.152744:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075954.152756:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075954.152778:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616475520:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075954.152780:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616475648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075954.152803:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769075954.152827:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075954.152832:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075954.152835:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075954.152893:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2939 00000100:00080000:1.0:1769075954.152897:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075954.152901:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075954.152905:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075954.152912:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000800:00000100:2.0:1769075954.152919:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075954.152923:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616475776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075954.152932:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075954.152935:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075954.152937:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075954.152939:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2939 00000100:00080000:1.0:1769075954.152943:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075954.152946:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075954.152950:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075954.152958:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075954/real 1769075954] req@ffff995b868c7100 x1855007616475520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075954.152962:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616475904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075954.153002:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075954.153005:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075954.153007:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075954.153010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616475520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075954.153016:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2939 00000100:00080000:1.0:1769075954.153024:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075954.153029:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075954.153030:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769075954.153038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769075954.153039:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075954.153046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616475520:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075954.153055:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616476032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075954.153067:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075954.153070:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075954.153072:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075954.153076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475648:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075954.153077:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2939 00000100:00080000:1.0:1769075954.153085:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075954.153089:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075954.153093:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075954.153105:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616476160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075954.153115:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075954.153120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075954.153121:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075954.153127:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075954.153132:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2939 00000100:00080000:1.0:1769075954.153136:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075954.153139:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075954.153144:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075954.153152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075954/real 1769075954] req@ffff995b868c6300 x1855007616475648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075954.153155:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616476288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075954.153222:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616475648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075954.153232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075954.153234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075954.153239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075954.153262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075954.153288:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075954.153301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075954/real 1769075954] req@ffff995b868c6d80 x1855007616475776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075954.153330:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616475776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075954.153377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075954.153386:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075954.153396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075954.153412:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475904:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075954.153441:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075954.153514:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075954/real 1769075954] req@ffff995b868c5c00 x1855007616475904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075954.153532:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616475904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075954.153543:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075954.153551:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075954.153554:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616475904:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075954.153559:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476032:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075954.153574:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075954.153583:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075954/real 1769075954] req@ffff995b868c4a80 x1855007616476032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075954.153591:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616476032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075954.153601:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075954.153606:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075954.153608:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476032:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075954.153621:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616476160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075954.153632:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075954.153639:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075954/real 1769075954] req@ffff995b868c7800 x1855007616476160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075954.153647:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616476160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075954.153652:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075954.153655:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075954.153658:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616476160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075954.153661:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616476288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075954.153703:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075954.153722:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075954/real 1769075954] req@ffff995b868c5500 x1855007616476288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075954.153739:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616476288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075954.153751:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075954.153756:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075954.153762:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616476288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075954.856521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075954.856749:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075955.176616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075955.176683:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075955.176701:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075955.176784:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075955.880593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075955.880838:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075956.200598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075956.200623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075956.200666:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075956.200678:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075956.904545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075956.904629:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075957.224555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075957.224596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075957.224604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075957.224619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075957.928545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075957.928638:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075958.248570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075958.248614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075958.248624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075958.248649:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075958.952521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075958.952612:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075959.272576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075959.272591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075959.272694:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075959.272806:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075959.272823:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075959.272831:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075959.272836:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075959.272849:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2944 00000100:00080000:2.0:1769075959.272888:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075959.272898:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075959.272955:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616476416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075959.272984:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075959.272987:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075959.272989:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075959.272993:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2944 00000100:00080000:2.0:1769075959.272996:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075959.272999:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075959.273010:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075959.273022:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616476544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075959.273034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616476416:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075959.273035:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075959.273047:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075959.273049:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075959.273053:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2944 00000100:00080000:2.0:1769075959.273056:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075959.273059:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075959.273063:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075959.273094:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616476672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075959.273103:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075959.273106:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075959.273107:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075959.273110:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2944 00000100:00080000:2.0:1769075959.273113:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075959.273117:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075959.273121:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769075959.273129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075959.273134:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616476800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075959.273145:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075959.273148:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075959.273151:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075959.273155:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2944 00000100:00080000:2.0:1769075959.273158:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075959.273161:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075959.273165:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075959.273177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075959/real 1769075959] req@ffff995b85ac9880 x1855007616476416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075959.273179:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616476928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075959.273222:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075959.273225:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075959.273227:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075959.273233:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2944 00000100:00100000:1.0:1769075959.273234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616476416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075959.273236:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075959.273244:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075959.273253:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075959.273260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769075959.273266:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616477056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075959.273270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075959.273281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616476416:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075959.273285:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075959.273290:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075959.273292:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075959.273297:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2944 00000100:00080000:2.0:1769075959.273299:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075959.273302:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075959.273309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476544:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769075959.273309:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075959.273354:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616477184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075959.273385:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075959.273409:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075959/real 1769075959] req@ffff995b85acaa00 x1855007616476544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075959.273427:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616476544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075959.273487:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075959.273492:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075959.273497:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075959.273505:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075959.273536:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075959.273557:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075959/real 1769075959] req@ffff995b85aca300 x1855007616476672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075959.273580:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616476672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075959.273600:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075959.273606:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075959.273613:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075959.273626:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476800:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075959.273660:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075959.273675:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075959/real 1769075959] req@ffff995b85acb800 x1855007616476800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075959.273689:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616476800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075959.273704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075959.273711:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075959.273718:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075959.273729:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476928:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075959.273791:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075959.273829:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075959/real 1769075959] req@ffff995b85ac9c00 x1855007616476928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075959.273847:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616476928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075959.273856:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075959.273860:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075959.273865:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616476928:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075959.273873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616477056:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075959.273903:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075959.273919:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075959/real 1769075959] req@ffff995b85acb100 x1855007616477056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075959.273937:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616477056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075959.273949:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075959.273954:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075959.273957:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616477056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075959.273976:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616477184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075959.273997:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075959.274007:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075959/real 1769075959] req@ffff995b85ac8380 x1855007616477184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075959.274017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616477184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075959.274025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075959.274027:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075959.274030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616477184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075959.976624:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075959.976770:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075960.296528:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075960.296550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075960.296557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075960.296583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075961.000526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075961.000612:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075961.320547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075961.320547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075961.320557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075961.320588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075962.024609:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075962.024753:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075962.344621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075962.344621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075962.344629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075962.344659:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075963.048512:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075963.048634:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075963.368551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075963.368554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075963.368558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075963.368589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075964.072601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075964.072754:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075964.392606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075964.392613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075964.392652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075964.392699:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075964.392736:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075964.392750:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075964.392757:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075964.392787:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2950 00000100:00080000:1.0:1769075964.392804:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075964.392816:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075964.392870:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616477312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075964.392910:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075964.392918:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075964.392921:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075964.392930:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2950 00000100:00080000:1.0:1769075964.392939:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075964.392948:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075964.392963:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075964.392979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616477312:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075964.392988:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616477440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075964.393006:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075964.393012:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075964.393015:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075964.393022:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2950 00000100:00080000:1.0:1769075964.393028:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075964.393034:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075964.393042:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075964.393059:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616477568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075964.393075:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075964.393076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075964.393081:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075964.393085:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075964.393091:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2950 00000100:00080000:1.0:1769075964.393096:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075964.393103:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075964.393111:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075964.393130:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616477696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075964.393137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075964/real 1769075964] req@ffff995b868c7480 x1855007616477312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075964.393162:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075964.393169:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769075964.393170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616477312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075964.393172:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075964.393179:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2950 00000100:00080000:1.0:1769075964.393186:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075964.393190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075964.393192:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769075964.393198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769075964.393200:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075964.393206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616477312:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075964.393216:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616477824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075964.393231:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075964.393237:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075964.393241:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075964.393242:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616477440:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769075964.393249:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2950 00000100:00080000:1.0:1769075964.393255:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075964.393262:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075964.393270:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075964.393281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075964.393287:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616477952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075964.393304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075964/real 1769075964] req@ffff995b868c4700 x1855007616477440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075964.393305:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075964.393312:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075964.393316:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075964.393323:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2950 00000100:00080000:1.0:1769075964.393330:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075964.393337:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075964.393345:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616477440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769075964.393346:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075964.393362:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616478080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075964.393362:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075964.393369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075964.393376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616477440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075964.393388:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616477568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075964.393427:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075964.393485:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075964/real 1769075964] req@ffff995b868c6a00 x1855007616477568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075964.393507:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616477568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075964.393520:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075964.393524:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075964.393529:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616477568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075964.393537:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616477696:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075964.393567:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075964.393582:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075964/real 1769075964] req@ffff995b868c5500 x1855007616477696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075964.393596:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616477696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075964.393615:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075964.393619:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075964.393623:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616477696:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075964.393629:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616477824:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075964.393647:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075964.393658:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075964/real 1769075964] req@ffff995b868c7800 x1855007616477824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075964.393704:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616477824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075964.393716:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075964.393720:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075964.393724:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616477824:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075964.393747:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616477952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075964.393770:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075964.393784:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075964/real 1769075964] req@ffff995b868c4a80 x1855007616477952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075964.393798:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616477952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075964.393808:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075964.393812:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075964.393816:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616477952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075964.393823:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616478080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075964.393841:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075964.393852:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075964/real 1769075964] req@ffff995b868c5c00 x1855007616478080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075964.393865:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616478080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075964.393876:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075964.393879:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075964.393882:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616478080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075965.096550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075965.096748:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075965.416627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075965.416642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075965.416730:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075965.416733:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075966.120586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075966.120785:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075966.440619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075966.440622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075966.440698:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075966.440736:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075967.144589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075967.144761:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075967.464583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075967.464583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075967.464607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075967.464646:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075968.168599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075968.168657:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075968.488701:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075968.488741:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075968.488812:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075968.488861:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075969.192628:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075969.192681:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075969.512690:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075969.512727:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075969.512760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075969.512771:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075969.512778:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769075969.512778:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075969.512779:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769075969.512802:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2955 00000100:00080000:2.0:1769075969.512817:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075969.512827:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075969.512872:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616478208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075969.512914:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075969.512919:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075969.512921:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075969.512926:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2955 00000100:00080000:2.0:1769075969.512930:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075969.512935:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075969.512969:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075969.512986:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616478336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075969.513003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075969.513007:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075969.513011:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075969.513020:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2955 00000100:00100000:1.0:1769075969.513025:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616478208:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075969.513027:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075969.513040:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075969.513055:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075969.513075:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616478464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075969.513091:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075969.513096:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075969.513099:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075969.513121:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2955 00000100:00080000:2.0:1769075969.513130:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075969.513136:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075969.513145:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075969.513157:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616478592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075969.513168:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075969.513173:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075969.513177:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769075969.513184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075969.513186:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2955 00000100:00080000:2.0:1769075969.513198:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075969.513202:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075969.513209:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075969.513220:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616478720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075969.513234:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075969.513240:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075969.513242:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075969.513246:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2955 00000100:00080000:2.0:1769075969.513251:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075969.513257:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075969.513263:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075969.513277:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616478848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075969.513285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075969/real 1769075969] req@ffff995b85acbb80 x1855007616478208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075969.513377:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075969.513382:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075969.513384:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075969.513401:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2955 00000100:00080000:2.0:1769075969.513405:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075969.513413:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616478208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075969.513414:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075969.513434:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769075969.513447:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769075969.513455:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616478976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075969.513456:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075969.513531:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616478208:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769075969.513621:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616478336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075969.513664:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075969.513682:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075969/real 1769075969] req@ffff995b85ac8a80 x1855007616478336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.513702:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616478336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.513722:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.513729:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075969.513737:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616478336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075969.513750:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616478464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075969.513792:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075969.513816:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075969/real 1769075969] req@ffff995b85ac8380 x1855007616478464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.513841:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616478464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.513856:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.513860:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075969.513865:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616478464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075969.513874:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616478592:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075969.513903:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075969.513916:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075969/real 1769075969] req@ffff995b85acb100 x1855007616478592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.513930:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616478592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.513951:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.513956:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075969.513960:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616478592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075969.513966:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616478720:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075969.513985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075969.513997:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075969/real 1769075969] req@ffff995b85ac9c00 x1855007616478720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.514011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616478720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.514023:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.514026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075969.514030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616478720:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075969.514037:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616478848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075969.514057:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075969.514068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075969/real 1769075969] req@ffff995b85acb800 x1855007616478848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.514083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616478848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.514093:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.514097:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075969.514101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616478848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075969.514108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616478976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075969.514126:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075969.514137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075969/real 1769075969] req@ffff995b85aca300 x1855007616478976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.514150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616478976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.514160:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.514164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075969.514168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616478976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075970.216665:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075970.216748:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075970.536586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075970.536644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075970.536650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075970.536700:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075971.240621:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075971.240759:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075971.560586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075971.560648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075971.560663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075971.560677:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075972.264612:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075972.264691:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075972.584611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075972.584685:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075972.584691:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075972.584691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075973.288611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075973.288737:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075973.608646:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075973.608647:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075973.608655:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075973.608659:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075974.312593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075974.312736:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075974.632572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075974.632572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075974.632584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769075974.632634:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075974.632639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075974.632643:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075974.632651:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2960 00000100:00100000:0.0:1769075974.632657:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769075974.632661:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075974.632672:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075974.632703:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616479104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075974.632725:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075974.632728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075974.632730:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075974.632733:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2960 00000100:00080000:1.0:1769075974.632735:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075974.632738:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075974.632751:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075974.632762:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616479232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075974.632764:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616479104:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075974.632778:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075974.632780:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075974.632782:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075974.632785:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2960 00000100:00080000:1.0:1769075974.632787:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075974.632790:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075974.632793:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075974.632801:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616479360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075974.632808:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075974.632810:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075974.632811:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075974.632814:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2960 00000100:00080000:1.0:1769075974.632817:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075974.632820:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075974.632823:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075974.632832:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616479488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075974.632838:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769075974.632844:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075974.632847:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075974.632849:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075974.632852:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2960 00000100:00080000:1.0:1769075974.632858:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075974.632860:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075974.632863:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075974.632871:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616479616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075974.632878:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075974/real 1769075974] req@ffff995b868c6d80 x1855007616479104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075974.632879:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075974.632886:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075974.632889:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075974.632897:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2960 00000100:00080000:1.0:1769075974.632900:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075974.632903:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075974.632907:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075974.632907:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616479104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075974.632919:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616479744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075974.632931:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075974.632939:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769075974.632944:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075974.632945:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616479104:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769075974.632949:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075974.632951:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075974.632955:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2960 00000100:00080000:1.0:1769075974.632957:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075974.632960:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769075974.632963:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075974.632971:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616479232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075974.632972:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616479872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075974.632995:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075974.633005:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075974/real 1769075974] req@ffff995b868c6300 x1855007616479232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075974.633014:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616479232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075974.633019:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075974.633022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075974.633024:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616479232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075974.633028:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616479360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075974.633037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075974.633043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075974/real 1769075974] req@ffff995b868c7100 x1855007616479360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075974.633050:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616479360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075974.633057:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075974.633059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075974.633061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616479360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075974.633065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616479488:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075974.633075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075974.633080:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075974/real 1769075974] req@ffff995b868c5c00 x1855007616479488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075974.633088:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616479488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075974.633093:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075974.633095:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075974.633097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616479488:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075974.633100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616479616:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075974.633109:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075974.633114:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075974/real 1769075974] req@ffff995b868c4a80 x1855007616479616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075974.633121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616479616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075974.633127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075974.633129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075974.633130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616479616:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075974.633134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616479744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075974.633143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075974.633148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075974/real 1769075974] req@ffff995b868c7800 x1855007616479744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075974.633156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616479744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075974.633161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075974.633163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075974.633165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616479744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075974.633175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616479872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075974.633184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075974.633190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075974/real 1769075974] req@ffff995b868c5500 x1855007616479872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075974.633196:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616479872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075974.633202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075974.633203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075974.633205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616479872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075975.336633:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075975.336680:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075975.656609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075975.656692:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075975.656771:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075975.656779:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075976.360601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075976.360704:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075976.680632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075976.680685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075976.680706:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075976.680728:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075977.384627:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075977.384763:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075977.704632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075977.704645:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075977.704645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075977.704741:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075978.408583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075978.408667:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075978.728572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075978.728583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075978.728650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075978.728655:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075979.432594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075979.432697:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075979.752632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075979.752638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769075979.752647:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075979.752660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769075979.752660:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075979.752672:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075979.752689:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2965 00000100:00080000:3.0:1769075979.752714:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075979.752720:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769075979.752729:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075979.752779:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616480000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075979.752817:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075979.752822:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075979.752825:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075979.752830:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2965 00000100:00080000:3.0:1769075979.752834:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075979.752840:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075979.752853:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075979.752873:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616480128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075979.752876:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616480000:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075979.752903:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075979.752908:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075979.752910:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075979.752916:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2965 00000100:00080000:3.0:1769075979.752921:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075979.752925:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075979.752930:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075979.752949:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616480256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075979.752970:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075979.752974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075979.752977:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769075979.752986:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769075979.752987:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2965 00000100:00080000:3.0:1769075979.753000:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075979.753019:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075979.753027:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769075979.753048:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616480384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075979.753059:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075979/real 1769075979] req@ffff995b8455b480 x1855007616480000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075979.753067:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075979.753081:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075979.753088:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075979.753108:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2965 00000100:00080000:3.0:1769075979.753114:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075979.753119:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075979.753124:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075979.753130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616480000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075979.753143:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616480512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075979.753188:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075979.753192:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075979.753194:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075979.753201:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075979.753212:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2965 00000100:00080000:2.0:1769075979.753219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769075979.753221:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075979.753235:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616480000:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769075979.753236:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075979.753250:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075979.753276:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616480640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075979.753292:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075979.753299:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769075979.753301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616480128:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769075979.753303:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075979.753317:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2965 00000100:00080000:3.0:1769075979.753321:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075979.753324:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769075979.753330:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075979.753346:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616480768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075979.753390:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075979.753452:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075979/real 1769075979] req@ffff995b8455ad80 x1855007616480128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075979.753505:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616480128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075979.753517:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075979.753522:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075979.753526:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616480128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075979.753535:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616480256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075979.753576:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075979.753592:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075979/real 1769075979] req@ffff995b84559880 x1855007616480256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075979.753607:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616480256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075979.753624:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075979.753628:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075979.753632:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616480256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075979.753640:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616480384:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075979.753659:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075979.753671:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075979/real 1769075979] req@ffff995b84559c00 x1855007616480384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075979.753685:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616480384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075979.753706:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075979.753709:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075979.753713:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616480384:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075979.753719:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616480512:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769075979.753737:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075979.753748:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075979/real 1769075979] req@ffff995b8455b100 x1855007616480512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075979.753762:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616480512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075979.753773:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075979.753778:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075979.753781:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616480512:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769075979.753803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616480640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075979.753821:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075979.753833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075979/real 1769075979] req@ffff995b8455a300 x1855007616480640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075979.753846:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616480640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075979.753857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075979.753860:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075979.753864:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616480640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075979.753870:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616480768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769075979.753887:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769075979.753898:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075979/real 1769075979] req@ffff995b84558700 x1855007616480768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075979.753912:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616480768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075979.753922:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075979.753925:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769075979.753930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616480768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075980.456608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075980.456684:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075980.776644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075980.776676:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075980.776701:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075980.776705:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075981.480581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075981.480681:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075981.800596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075981.800626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075981.800659:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075981.800668:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075982.504538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075982.504603:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075982.824605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075982.824619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075982.824687:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075982.824697:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075983.528589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075983.528815:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075983.848616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075983.848645:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075983.848689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075983.848694:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075984.552555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075984.552701:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075984.872621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075984.872640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075984.872677:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769075984.872687:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769075984.872693:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075984.872702:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075984.872705:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075984.872718:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2970 00000100:00080000:2.0:1769075984.872739:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075984.872753:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769075984.872813:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616480896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075984.872857:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075984.872863:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075984.872866:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075984.872872:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2970 00000100:00080000:2.0:1769075984.872878:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075984.872886:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075984.872902:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075984.872920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616480896:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075984.872927:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616481024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075984.872945:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075984.872950:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075984.872952:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075984.872956:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2970 00000100:00080000:2.0:1769075984.872960:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075984.872964:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075984.872970:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075984.873003:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616481152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075984.873016:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075984.873020:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075984.873022:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075984.873030:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2970 00000100:00080000:2.0:1769075984.873035:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075984.873044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075984.873046:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075984.873059:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075984.873077:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616481280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075984.873089:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075984.873094:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075984.873097:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075984.873103:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2970 00000100:00080000:2.0:1769075984.873108:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075984.873111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075984/real 1769075984] req@ffff995b85acaa00 x1855007616480896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075984.873118:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075984.873134:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075984.873151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616480896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075984.873166:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616481408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075984.873171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075984.873190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075984.873202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616480896:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075984.873222:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075984.873230:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075984.873235:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075984.873243:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481024:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075984.873248:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2970 00000100:00080000:2.0:1769075984.873264:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075984.873273:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075984.873282:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075984.873284:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075984.873316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075984/real 1769075984] req@ffff995b85ac9880 x1855007616481024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075984.873347:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616481536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075984.873393:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075984.873398:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616481024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075984.873401:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075984.873408:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075984.873424:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2970 00000100:00080000:1.0:1769075984.873430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075984.873433:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075984.873442:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769075984.873446:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075984.873488:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481024:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769075984.873501:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075984.873506:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075984.873532:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616481664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075984.873551:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075984.873574:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075984/real 1769075984] req@ffff995b85aca300 x1855007616481152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075984.873600:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616481152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075984.873613:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075984.873617:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075984.873621:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075984.873628:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481280:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075984.873649:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075984.873660:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075984/real 1769075984] req@ffff995b85acb800 x1855007616481280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075984.873680:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616481280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075984.873699:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075984.873706:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075984.873713:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075984.873724:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481408:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075984.873763:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075984.873786:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075984/real 1769075984] req@ffff995b85ac9c00 x1855007616481408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075984.873810:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616481408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075984.873840:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075984.873847:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075984.873853:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075984.873889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616481536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075984.873921:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075984.873937:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075984/real 1769075984] req@ffff995b85acb100 x1855007616481536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075984.873952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616481536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075984.873964:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075984.873968:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075984.873972:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616481536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075984.873979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616481664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075984.873997:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075984.874008:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075984/real 1769075984] req@ffff995b85ac8380 x1855007616481664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075984.874022:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616481664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075984.874031:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075984.874035:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075984.874040:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616481664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075985.576597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075985.576706:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075985.896604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075985.896632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075985.896658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075985.896696:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075986.600560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075986.600701:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075986.920619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075986.920656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769075986.920846:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075986.920875:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075987.624576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075987.624704:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075987.944601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075987.944629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769075987.944684:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075987.944690:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075988.648567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075988.648686:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075988.968639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075988.968665:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075988.968714:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075988.968716:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075989.672579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075989.672705:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075989.992596:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075989.992610:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075989.992617:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075989.992633:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2975 00000100:00080000:2.0:1769075989.992650:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075989.992661:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769075989.992669:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769075989.992678:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075989.992705:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075989.992729:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616481792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075989.992768:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075989.992774:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075989.992777:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075989.992790:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2975 00000100:00100000:3.0:1769075989.992798:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769075989.992799:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075989.992812:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075989.992826:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075989.992855:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616481920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075989.992857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616481792:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075989.992907:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075989.992914:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075989.992917:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075989.992929:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2975 00000100:00080000:2.0:1769075989.992938:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075989.992947:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075989.992961:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075989.993012:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616482048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075989.993040:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075989.993051:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075989.993052:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075989.993059:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075989.993068:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2975 00000100:00080000:2.0:1769075989.993076:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075989.993085:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075989.993097:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075989.993126:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616482176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075989.993145:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075989.993151:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075989.993157:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075989.993169:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2975 00000100:00080000:2.0:1769075989.993178:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075989.993190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075989/real 1769075989] req@ffff995b85ac8a80 x1855007616481792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075989.993191:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075989.993213:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075989.993284:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616482304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075989.993291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616481792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075989.993332:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075989.993349:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075989.993363:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075989.993373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075989.993375:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2975 00000100:00080000:2.0:1769075989.993395:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075989.993406:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769075989.993407:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075989.993428:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075989.993443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616481792:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075989.993515:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616482432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075989.993533:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075989.993538:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075989.993542:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075989.993550:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2975 00000100:00080000:2.0:1769075989.993556:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075989.993566:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481920:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075989.993567:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075989.993589:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075989.993616:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616482560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075989.993659:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075989.993704:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075989/real 1769075989] req@ffff995b85acbb80 x1855007616481920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075989.993740:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616481920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075989.993760:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075989.993772:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075989.993782:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616481920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075989.993808:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075989.993856:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075989.993879:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075989/real 1769075989] req@ffff995b85ac8380 x1855007616482048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075989.993896:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616482048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075989.993911:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075989.993916:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075989.993921:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075989.993930:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482176:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075989.993957:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075989.993969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075989/real 1769075989] req@ffff995b85acb100 x1855007616482176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075989.993984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616482176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075989.993996:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075989.994000:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075989.994005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482176:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075989.994013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482304:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075989.994032:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075989.994043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075989/real 1769075989] req@ffff995b85ac9c00 x1855007616482304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075989.994057:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616482304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075989.994070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075989.994074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075989.994077:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482304:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075989.994085:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616482432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075989.994104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075989.994116:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075989/real 1769075989] req@ffff995b85acb800 x1855007616482432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075989.994130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616482432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075989.994141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075989.994145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075989.994148:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616482432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075989.994176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616482560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075989.994197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075989.994210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075989/real 1769075989] req@ffff995b85aca300 x1855007616482560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075989.994225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616482560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075989.994235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075989.994271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075989.994275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616482560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075990.696500:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075990.696651:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075991.016546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075991.016551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075991.016612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075991.016624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769075991.720589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075991.720671:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075992.040605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075992.040617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075992.040703:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075992.040732:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075992.744589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075992.744706:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075993.064588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769075993.064639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075993.064644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769075993.064698:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075993.768596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075993.768686:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075994.088609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075994.088615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769075994.088620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075994.088637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075994.792571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075994.792676:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075995.112630:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075995.112631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075995.112631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769075995.112647:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075995.112655:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075995.112658:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769075995.112673:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2980 00000100:00080000:2.0:1769075995.112691:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075995.112704:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769075995.112714:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075995.112764:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616482688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075995.112804:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075995.112813:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075995.112819:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075995.112829:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2980 00000100:00080000:2.0:1769075995.112836:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075995.112845:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075995.112859:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075995.112862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616482688:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769075995.112895:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616482816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075995.112914:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075995.112918:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075995.112921:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075995.112926:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2980 00000100:00080000:2.0:1769075995.112933:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075995.112937:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075995.112943:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075995.112959:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616482944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075995.112971:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075995.112978:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075995.112980:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075995.112987:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075995.112994:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2980 00000100:00080000:2.0:1769075995.113000:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075995.113006:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075995.113014:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769075995.113029:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616483072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075995.113042:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075995.113049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075995/real 1769075995] req@ffff995b85ac9880 x1855007616482688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075995.113050:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075995.113058:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075995.113072:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2980 00000100:00080000:2.0:1769075995.113083:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075995.113090:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075995.113097:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769075995.113102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616482688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075995.113118:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616483200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075995.113124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075995.113142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769075995.113149:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075995.113188:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616482688:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769075995.113206:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075995.113212:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075995.113224:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2980 00000100:00100000:1.0:1769075995.113231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482816:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769075995.113235:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075995.113252:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075995.113261:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075995.113281:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616483328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075995.113294:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769075995.113306:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075995.113314:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075995.113318:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075995.113328:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2980 00000100:00100000:1.0:1769075995.113329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075995/real 1769075995] req@ffff995b85acaa00 x1855007616482816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075995.113337:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075995.113351:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769075995.113367:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075995.113375:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616482816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075995.113391:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616483456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075995.113395:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075995.113412:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075995.113426:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075995.113439:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075995.113503:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075995.113518:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075995/real 1769075995] req@ffff995b85aca300 x1855007616482944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075995.113534:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616482944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075995.113547:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075995.113552:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075995.113556:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616482944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075995.113564:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483072:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075995.113585:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075995.113597:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075995/real 1769075995] req@ffff995b85acb800 x1855007616483072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075995.113621:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616483072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075995.113632:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075995.113636:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075995.113640:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075995.113647:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483200:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769075995.113664:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075995.113675:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075995/real 1769075995] req@ffff995b85ac9c00 x1855007616483200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075995.113689:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616483200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075995.113700:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075995.113704:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075995.113707:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483200:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769075995.113728:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616483328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075995.113747:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075995.113759:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075995/real 1769075995] req@ffff995b85acb100 x1855007616483328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075995.113773:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616483328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075995.113792:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075995.113796:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075995.113799:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616483328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769075995.113806:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616483456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769075995.113822:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769075995.113833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075995/real 1769075995] req@ffff995b85ac8380 x1855007616483456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075995.113847:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616483456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075995.113857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075995.113860:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769075995.113864:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616483456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769075995.816612:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075995.816717:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075996.136588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769075996.136606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075996.136643:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075996.136693:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075996.840573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075996.840679:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075997.160590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075997.160652:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769075997.160743:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769075997.160790:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075997.864599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075997.864696:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075998.184584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769075998.184644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075998.184678:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075998.184699:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075998.888545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075998.888713:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075999.208543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769075999.208565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075999.208620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769075999.208627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769075999.912540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075999.912651:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076000.232580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076000.232585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076000.232619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076000.232656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076000.232657:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076000.232666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076000.232671:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076000.232682:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2985 00000100:00080000:3.0:1769076000.232697:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076000.232708:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076000.232757:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616483584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076000.232804:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076000.232809:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076000.232811:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076000.232817:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2985 00000100:00080000:3.0:1769076000.232823:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076000.232828:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076000.232839:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076000.232845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616483584:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076000.232859:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616483712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076000.232903:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076000.232907:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076000.232909:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076000.232914:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2985 00000100:00080000:3.0:1769076000.232920:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076000.232925:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076000.232931:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076000.232944:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616483840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076000.232972:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076000.232976:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076000.232978:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076000.232981:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2985 00000100:00080000:3.0:1769076000.232985:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076000.232989:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076000.232996:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076000.233007:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616483968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076000.233016:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076000.233019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076000.233022:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076000.233029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483712:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076000.233032:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2985 00000100:00080000:3.0:1769076000.233040:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076000.233045:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076000.233052:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076000.233085:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616484096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076000.233101:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076000.233107:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769076000.233110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483840:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076000.233111:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076000.233129:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2985 00000100:00080000:3.0:1769076000.233138:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076000.233143:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076000.233150:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076000.233165:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616484224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076000.233176:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076000.233180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483968:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076000.233181:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076000.233196:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076000.233205:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2985 00000100:00080000:3.0:1769076000.233209:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076000.233213:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076000.233220:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076000.233236:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616484352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076000.233242:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484096:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076000.233281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616484224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076000.233304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616484352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076000.936610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076000.936719:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076001.256599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076001.256662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076001.256691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076001.256698:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076001.960599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076001.960680:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076002.280568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076002.280635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076002.280653:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076002.280726:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076002.984561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076002.984670:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076003.304600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076003.304609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076003.304717:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076003.304798:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:0.0:1769076003.304930:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:0.0:1769076003.304997:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769076003.305052:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769076003.305144:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769076003.305164:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769076003.305179:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769076003.305205:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769076003.305218:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769076003.305232:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:1.0:1769076003.305374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076000/real 1769076003] req@ffff995b8455b480 x1855007616484352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076003.305426:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616484352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076003.305447:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076003.305634:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076003.305645:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616484352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076003.305661:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076000/real 1769076003] req@ffff995b84559880 x1855007616484096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076003.305692:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616484096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076003.305711:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076003.305719:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076003.305726:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484096:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076003.305739:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076000/real 1769076003] req@ffff995b8455ad80 x1855007616484224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076003.305763:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616484224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076003.305780:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076003.305787:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076003.305794:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616484224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076003.305809:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076000/real 1769076003] req@ffff995b8455a300 x1855007616483712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076003.305835:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616483712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076003.305850:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076003.305858:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076003.305864:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076003.305876:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076000/real 1769076003] req@ffff995b8455b100 x1855007616483840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076003.305899:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616483840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076003.305918:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076003.305924:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076003.305930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076003.305942:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076000/real 1769076003] req@ffff995b84559c00 x1855007616483968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076003.305964:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616483968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076003.305980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076003.305988:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076003.305994:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616483968:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076003.306008:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076000/real 1769076003] req@ffff995b84558700 x1855007616483584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076003.306058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616483584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076003.306074:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076003.306081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076003.306089:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616483584:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076004.008592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076004.008769:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076004.328646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076004.328698:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076004.328748:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076004.328778:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076005.032578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076005.032682:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076005.352656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076005.352664:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076005.352712:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076005.352720:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076005.352767:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076005.352779:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076005.352785:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076005.352797:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2991 00000100:00080000:0.0:1769076005.352810:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076005.352819:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076005.352862:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616484480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076005.352899:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076005.352904:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076005.352906:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076005.352912:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2991 00000100:00080000:0.0:1769076005.352915:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076005.352920:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076005.352931:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076005.352945:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616484608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076005.352958:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076005.352963:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076005.352967:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076005.352975:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2991 00000100:00080000:0.0:1769076005.353005:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076005.353005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616484480:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076005.353021:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076005.353032:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076005.353057:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616484736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076005.353077:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076005.353084:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076005.353088:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076005.353094:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2991 00000100:00080000:0.0:1769076005.353101:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076005.353106:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076005.353114:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076005.353126:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076005.353148:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616484864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076005.353165:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076005.353171:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076005.353180:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076005.353180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076005/real 1769076005] req@ffff995b84642a00 x1855007616484480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076005.353194:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2991 00000100:00080000:0.0:1769076005.353206:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076005.353218:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076005.353229:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076005.353240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616484480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076005.353257:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616484992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076005.353268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076005.353284:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076005.353286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076005.353299:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769076005.353302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616484480:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076005.353305:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076005.353323:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2991 00000100:00080000:0.0:1769076005.353332:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076005.353340:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484608:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076005.353343:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076005.353362:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076005.353390:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616485120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076005.353407:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076005.353408:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076005.353419:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076005.353422:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076005.353431:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2991 00000100:00080000:0.0:1769076005.353438:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076005.353446:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076005.353494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076005/real 1769076005] req@ffff995b84640e00 x1855007616484608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076005.353521:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616484608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076005.353522:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769076005.353546:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769076005.353556:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616485248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076005.353561:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076005.353577:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076005.353627:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076005.353673:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076005.353695:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076005/real 1769076005] req@ffff995b84641180 x1855007616484736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076005.353720:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616484736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076005.353738:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076005.353745:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076005.353752:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076005.353766:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484864:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076005.353809:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076005.353830:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076005/real 1769076005] req@ffff995b84640000 x1855007616484864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076005.353851:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616484864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076005.353867:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076005.353873:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076005.353879:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484864:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076005.353890:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484992:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076005.353925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076005.353945:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076005/real 1769076005] req@ffff995b84641c00 x1855007616484992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076005.353977:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616484992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076005.354018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076005.354024:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076005.354030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616484992:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076005.354041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616485120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076005.354078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076005.354097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076005/real 1769076005] req@ffff995b84642680 x1855007616485120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076005.354128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616485120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076005.354142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076005.354147:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076005.354153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616485120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076005.354164:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616485248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076005.354197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076005.354216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076005/real 1769076005] req@ffff995b84640700 x1855007616485248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076005.354233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616485248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076005.354255:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076005.354260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076005.354264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616485248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076006.056579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076006.056702:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076006.376657:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076006.376658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076006.376717:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076006.376807:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076007.080596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076007.080653:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076007.400597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076007.400641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076007.400655:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076007.400660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076008.104569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076008.104702:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076008.424656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076008.424664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076008.424670:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076008.424678:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076009.128579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076009.128732:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076009.448571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076009.448572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076009.448696:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076009.448703:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076010.152594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076010.152712:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076010.472716:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076010.472722:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076010.472723:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076010.472746:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076010.472756:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076010.472778:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076010.472791:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 2996 00000100:00100000:2.0:1769076010.472803:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076010.472806:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076010.472821:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076010.472880:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616485376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076010.472930:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076010.472936:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076010.472954:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076010.472960:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2996 00000100:00080000:1.0:1769076010.472968:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076010.472976:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076010.472989:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076010.473013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616485376:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076010.473018:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616485504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076010.473058:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076010.473062:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076010.473064:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076010.473072:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2996 00000100:00080000:1.0:1769076010.473077:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076010.473081:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076010.473088:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076010.473102:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616485632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076010.473111:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076010.473114:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076010.473116:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076010.473119:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2996 00000100:00080000:1.0:1769076010.473125:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076010.473130:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076010.473135:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076010.473148:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616485760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076010.473155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076010.473174:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076010.473178:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076010.473180:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076010.473189:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 2996 00000100:00080000:1.0:1769076010.473203:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076010.473206:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076010.473211:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076010.473224:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616485888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076010.473235:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076010.473244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076010/real 1769076010] req@ffff995b868c6a00 x1855007616485376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076010.473245:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076010.473251:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076010.473293:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2996 00000100:00080000:1.0:1769076010.473300:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076010.473301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616485376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076010.473327:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076010.473333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769076010.473343:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076010.473345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076010.473364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616485376:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076010.473400:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616486016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076010.473411:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616485504:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076010.473419:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076010.473425:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076010.473428:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076010.473436:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 2996 00000100:00080000:1.0:1769076010.473444:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076010.473450:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076010.473501:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769076010.473511:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076010.473525:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076010/real 1769076010] req@ffff995b868c4700 x1855007616485504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076010.473539:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616486144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076010.473569:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616485504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076010.473599:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076010.473605:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076010.473610:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616485504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076010.473623:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616485632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076010.473648:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076010.473660:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076010/real 1769076010] req@ffff995b868c7480 x1855007616485632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076010.473684:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616485632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076010.473697:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076010.473701:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076010.473705:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616485632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076010.473713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616485760:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076010.473733:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076010.473744:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076010/real 1769076010] req@ffff995b868c5500 x1855007616485760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076010.473758:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616485760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076010.473777:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076010.473781:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076010.473784:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616485760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076010.473791:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616485888:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076010.473808:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076010.473820:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076010/real 1769076010] req@ffff995b868c7800 x1855007616485888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076010.473834:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616485888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076010.473853:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076010.473858:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076010.473861:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616485888:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076010.473883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616486016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076010.473933:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076010.473947:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076010/real 1769076010] req@ffff995b868c4a80 x1855007616486016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076010.473962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616486016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076010.473973:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076010.473977:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076010.473981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616486016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076010.473988:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616486144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076010.474007:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076010.474017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076010/real 1769076010] req@ffff995b868c5c00 x1855007616486144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076010.474031:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616486144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076010.474040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076010.474044:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076010.474048:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616486144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076011.176552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076011.176698:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076011.496665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076011.496670:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076011.496697:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076011.496748:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076012.200532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076012.200725:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076012.520730:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076012.520746:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076012.520773:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076012.520818:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076013.224573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076013.224682:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076013.544541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076013.544562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076013.544574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076013.545542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076014.248551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076014.248699:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076014.568600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076014.568674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076014.568675:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076014.568689:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076015.272564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076015.272647:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076015.592607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076015.592608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076015.592672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076015.592691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076015.592700:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076015.592705:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076015.592718:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3001 00000100:00080000:3.0:1769076015.592733:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076015.592742:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076015.592783:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616486272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076015.592886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076015.592891:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076015.592894:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076015.592900:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3001 00000100:00080000:3.0:1769076015.592905:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076015.592910:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076015.592920:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076015.592938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616486400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076015.592948:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616486272:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076015.592956:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076015.592962:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076015.592965:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076015.592972:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3001 00000100:00080000:3.0:1769076015.592979:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076015.592986:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076015.592997:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076015.593020:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616486528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076015.593038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076015.593045:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076015.593054:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076015.593059:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076015.593068:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3001 00000100:00080000:3.0:1769076015.593077:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076015.593084:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076015.593090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076015/real 1769076015] req@ffff995b84558000 x1855007616486272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076015.593095:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076015.593141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616486272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076015.593151:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616486656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076015.593160:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076015.593174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076015.593176:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076015.593184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616486272:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076015.593186:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076015.593194:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076015.593226:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3001 00000100:00080000:3.0:1769076015.593251:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076015.593258:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076015.593265:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616486400:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076015.593270:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076015.593304:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616486784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076015.593327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076015.593341:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076015.593350:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076015.593354:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076015.593361:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3001 00000100:00100000:2.0:1769076015.593366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076015/real 1769076015] req@ffff995b8455a680 x1855007616486400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076015.593370:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076015.593387:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076015.593407:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076015.593411:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616486400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076015.593425:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076015.593429:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076015.593435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616486400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076015.593443:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616486912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076015.593450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616486528:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076015.593465:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076015.593537:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076015.593542:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076015.593551:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076015.593559:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3001 00000100:00080000:3.0:1769076015.593570:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076015.593572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076015.593579:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076015.593590:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076015.593598:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076015/real 1769076015] req@ffff995b84558a80 x1855007616486528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076015.593617:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616487040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076015.593622:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616486528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076015.593665:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076015.593671:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076015.593676:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616486528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076015.593685:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616486656:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076015.593709:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076015.593720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076015/real 1769076015] req@ffff995b84559500 x1855007616486656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076015.593733:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616486656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076015.593751:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076015.593755:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076015.593759:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616486656:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076015.593780:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616486784:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076015.593832:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076015.593856:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076015/real 1769076015] req@ffff995b8455aa00 x1855007616486784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076015.593887:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616486784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076015.593904:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076015.593910:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076015.593917:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616486784:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076015.593927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616486912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076015.593956:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076015.593969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076015/real 1769076015] req@ffff995b8455bb80 x1855007616486912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076015.594010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616486912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076015.594020:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076015.594023:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076015.594027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616486912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076015.594033:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616487040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076015.594050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076015.594060:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076015/real 1769076015] req@ffff995b84558000 x1855007616487040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076015.594072:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616487040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076015.594081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076015.594084:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076015.594087:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616487040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076016.296568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076016.296694:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076016.616623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076016.616646:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076016.616686:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076016.617526:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076017.320562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076017.320665:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076017.640578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076017.640658:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076017.640737:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076017.640744:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076018.344593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076018.344699:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076018.664570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076018.664612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076018.664657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076018.664694:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076019.368581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076019.368691:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076019.688598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076019.688619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076019.688643:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076019.688673:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076020.392579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076020.392688:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076020.712530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076020.712542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076020.712549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076020.712562:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076020.712567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769076020.712568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076020.712570:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076020.712580:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3006 00000100:00080000:0.0:1769076020.712590:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076020.712595:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076020.712625:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616487168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076020.712655:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076020.712658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076020.712660:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076020.712664:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3006 00000100:00080000:0.0:1769076020.712667:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076020.712670:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076020.712676:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616487168:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076020.712677:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076020.712690:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616487296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076020.712698:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076020.712701:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076020.712702:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076020.712705:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3006 00000100:00080000:0.0:1769076020.712708:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076020.712711:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076020.712714:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076020.712734:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616487424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076020.712742:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076020.712744:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076020.712745:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076020.712748:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3006 00000100:00080000:0.0:1769076020.712754:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076020.712756:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076020.712760:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076020.712774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076020.712782:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616487552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076020.712790:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076020.712792:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076020.712794:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076020.712796:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3006 00000100:00080000:0.0:1769076020.712798:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076020.712800:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076020.712803:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076020.712812:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076020/real 1769076020] req@ffff995b84640380 x1855007616487168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076020.712814:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616487680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076020.712845:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076020.712858:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076020.712860:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076020.712864:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616487168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076020.712867:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3006 00000100:00080000:2.0:1769076020.712875:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076020.712876:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076020.712883:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076020.712886:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076020.712889:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616487168:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076020.712895:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076020.712910:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616487808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076020.712912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616487296:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076020.712929:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076020.712933:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1769076020.712937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076020.712938:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076020.712946:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3006 00000100:00080000:0.0:1769076020.712950:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076020.712951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076020/real 1769076020] req@ffff995b84641f80 x1855007616487296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076020.712958:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076020.712968:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076020.712968:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616487296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076020.712987:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076020.712989:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076020.712992:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616487296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076020.712999:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616487936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076020.712999:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616487424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076020.713018:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076020.713028:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076020/real 1769076020] req@ffff995b84640a80 x1855007616487424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076020.713036:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616487424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076020.713048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076020.713050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076020.713053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616487424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076020.713056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616487552:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076020.713077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076020.713083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076020/real 1769076020] req@ffff995b84640700 x1855007616487552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076020.713090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616487552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076020.713096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076020.713098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076020.713100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616487552:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076020.713103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616487680:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076020.713112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076020.713118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076020/real 1769076020] req@ffff995b84642680 x1855007616487680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076020.713125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616487680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076020.713131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076020.713133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076020.713135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616487680:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076020.713147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616487808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076020.713157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076020.713162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076020/real 1769076020] req@ffff995b84641c00 x1855007616487808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076020.713169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616487808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076020.713175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076020.713177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076020.713179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616487808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076020.713182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616487936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076020.713191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076020.713204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076020/real 1769076020] req@ffff995b84640000 x1855007616487936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076020.713213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616487936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076020.713218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076020.713221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076020.713223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616487936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076021.416553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076021.416707:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076021.736633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076021.736634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076021.736634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076021.736635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076022.440575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076022.440773:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076022.760565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076022.760596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076022.760611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076022.760619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076023.464624:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076023.464784:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076023.784606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076023.784630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076023.784638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076023.784707:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076024.488607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076024.488749:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076024.808591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076024.808610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076024.808632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076024.808718:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076025.512615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076025.512738:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076025.832679:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076025.832693:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076025.832745:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076025.832763:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076025.832770:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076025.832773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076025.832782:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076025.832794:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3011 00000100:00080000:3.0:1769076025.832808:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076025.832820:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076025.832866:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616488064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076025.832914:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076025.832920:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076025.832923:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076025.832930:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3011 00000100:00080000:3.0:1769076025.832935:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076025.832942:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076025.832953:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616488064:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769076025.832959:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076025.833000:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616488192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076025.833020:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076025.833026:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076025.833030:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076025.833035:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3011 00000100:00080000:3.0:1769076025.833043:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076025.833049:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076025.833056:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076025.833083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076025.833093:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616488320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076025.833110:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076025.833115:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076025.833118:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076025.833125:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3011 00000100:00100000:0.0:1769076025.833176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076025/real 1769076025] req@ffff995b84558700 x1855007616488064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076025.833205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616488064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076025.833220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076025.833227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076025.833234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616488064:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076025.833239:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076025.833252:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076025.833263:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076025.833272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616488192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076025.833307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076025.833324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076025/real 1769076025] req@ffff995b84559c00 x1855007616488192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076025.833340:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616488192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076025.833351:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076025.833355:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076025.833360:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616488192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076025.833369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616488320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076025.833404:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076025.833417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076025/real 1769076025] req@ffff995b8455b100 x1855007616488320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076025.833433:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616488320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076025.833509:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076025.833519:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616488448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076025.833528:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076025.833556:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616488320:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076025.833628:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076025.833635:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076025.833662:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076025.833669:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3011 00000100:00080000:3.0:1769076025.833689:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076025.833700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616488448:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076025.833702:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076025.833719:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076025.833746:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616488576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076025.833766:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076025.833771:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076025.833774:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076025.833784:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076025.833793:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3011 00000100:00080000:3.0:1769076025.833802:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076025.833809:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076025.833817:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076025.833823:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076025/real 1769076025] req@ffff995b8455a300 x1855007616488448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076025.833843:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616488704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076025.833856:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616488448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076025.833868:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076025.833879:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076025.833881:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076025.833883:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076025.833897:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076025.833902:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3011 00000100:00100000:2.0:1769076025.833911:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616488448:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076025.833913:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076025.833928:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076025.833936:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076025.833947:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616488576:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076025.833956:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616488832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076025.833990:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076025.834009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076025/real 1769076025] req@ffff995b8455ad80 x1855007616488576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076025.834027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616488576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076025.834038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076025.834043:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076025.834047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616488576:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076025.834055:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616488704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076025.834074:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076025.834086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076025/real 1769076025] req@ffff995b84559880 x1855007616488704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076025.834100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616488704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076025.834110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076025.834114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076025.834117:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616488704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076025.834132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616488832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076025.834150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076025.834162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076025/real 1769076025] req@ffff995b8455b480 x1855007616488832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076025.834176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616488832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076025.834186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076025.834189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076025.834193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616488832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076026.536552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076026.536703:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076026.856514:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076026.856533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076026.856536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076026.856595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076027.560657:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076027.560712:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076027.880738:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076027.880747:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076027.880794:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076027.880808:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076028.584636:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076028.584737:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076028.904619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076028.904647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076028.904711:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076028.904723:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076029.608650:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076029.608765:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076029.928678:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076029.928701:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076029.928738:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076029.928751:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076030.632577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076030.632755:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076030.952601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076030.952636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076030.952679:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076030.952689:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076030.952696:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076030.952714:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076030.952714:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3016 00000100:00100000:3.0:1769076030.952724:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076030.952735:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076030.952756:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076030.952829:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616488960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076030.952865:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076030.952870:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076030.952874:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076030.952881:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3016 00000100:00080000:2.0:1769076030.952887:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076030.952893:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076030.952909:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076030.952928:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616489088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076030.952953:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076030.952961:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076030.952965:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616488960:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076030.952965:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076030.952993:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3016 00000100:00080000:2.0:1769076030.952999:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076030.953004:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076030.953012:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076030.953042:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616489216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076030.953055:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076030.953061:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076030.953065:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076030.953070:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3016 00000100:00080000:2.0:1769076030.953074:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076030.953080:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076030.953089:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076030.953104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076030.953113:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616489344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076030.953134:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076030.953139:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076030.953142:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076030.953147:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3016 00000100:00080000:2.0:1769076030.953153:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076030.953161:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076030.953172:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076030.953181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076030/real 1769076030] req@ffff995b85acbb80 x1855007616488960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076030.953193:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616489472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076030.953252:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076030.953256:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076030.953259:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076030.953268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616488960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076030.953277:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3016 00000100:00080000:2.0:1769076030.953292:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076030.953309:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076030.953315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076030.953319:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076030.953328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076030.953343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616488960:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076030.953344:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616489600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076030.953381:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076030.953387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076030.953391:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076030.953397:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3016 00000100:00080000:2.0:1769076030.953404:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076030.953414:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489088:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076030.953415:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076030.953436:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076030.953514:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076030.953518:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616489728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076030.953546:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076030/real 1769076030] req@ffff995b85ac8a80 x1855007616489088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076030.953628:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616489088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076030.953651:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076030.953660:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076030.953668:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076030.953683:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076030.953729:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076030.953754:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076030/real 1769076030] req@ffff995b85ac8380 x1855007616489216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076030.953771:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616489216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076030.953785:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076030.953789:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076030.953794:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076030.953807:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489344:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076030.953847:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076030.953871:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076030/real 1769076030] req@ffff995b85acb100 x1855007616489344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076030.953897:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616489344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076030.953915:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076030.953922:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076030.953928:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489344:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076030.953937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489472:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076030.953966:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076030.953981:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076030/real 1769076030] req@ffff995b85ac9c00 x1855007616489472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076030.953996:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616489472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076030.954007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076030.954011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076030.954015:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489472:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076030.954022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616489600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076030.954041:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076030.954052:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076030/real 1769076030] req@ffff995b85acb800 x1855007616489600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076030.954066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616489600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076030.954076:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076030.954080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076030.954083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616489600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076030.954114:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616489728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076030.954134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076030.954147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076030/real 1769076030] req@ffff995b85aca300 x1855007616489728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076030.954161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616489728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076030.954172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076030.954175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076030.954179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616489728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076031.656618:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076031.656715:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076031.976611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076031.976675:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076031.976680:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076031.976688:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076032.680623:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076032.680716:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076033.000636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076033.000642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076033.000686:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076033.000735:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076033.704638:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076033.704754:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076034.024694:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076034.024697:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076034.024708:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076034.024769:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076034.728598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076034.728661:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076035.048571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076035.048575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076035.048639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076035.048677:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076035.752602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076035.752639:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076036.072652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076036.072688:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076036.072695:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076036.072706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076036.072714:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076036.072731:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3021 00000100:00100000:1.0:1769076036.072743:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076036.072746:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076036.072762:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076036.072812:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076036.072826:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616489856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076036.072879:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076036.072888:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076036.072892:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076036.072900:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3021 00000100:00080000:0.0:1769076036.072907:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076036.072916:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076036.072929:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076036.072934:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616489856:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076036.072956:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616489984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076036.072974:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076036.072979:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076036.072983:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076036.072988:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3021 00000100:00080000:0.0:1769076036.072992:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076036.072997:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076036.073003:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076036.073019:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616490112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076036.073035:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076036.073037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076036.073046:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076036.073049:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076036.073057:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3021 00000100:00080000:0.0:1769076036.073086:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076036.073092:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076036.073103:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076036.073109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076036/real 1769076036] req@ffff995b84641180 x1855007616489856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076036.073133:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616490240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076036.073168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616489856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076036.073169:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076036.073191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076036.073201:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076036.073203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076036.073215:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3021 00000100:00080000:2.0:1769076036.073216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076036.073229:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076036.073231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616489856:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076036.073246:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076036.073254:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076036.073276:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616490368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076036.073292:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076036.073300:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489984:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076036.073301:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076036.073314:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076036.073323:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3021 00000100:00080000:0.0:1769076036.073330:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076036.073335:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076036.073342:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076036.073369:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076036.073378:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616490496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076036.073393:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076036.073398:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076036.073401:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076036.073406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076036/real 1769076036] req@ffff995b84640e00 x1855007616489984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076036.073412:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3021 00000100:00080000:0.0:1769076036.073428:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076036.073443:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076036.073448:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616489984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076036.073534:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076036.073541:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076036.073550:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076036.073556:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616489984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076036.073570:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616490624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076036.073572:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616490112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076036.073627:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076036.073648:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076036/real 1769076036] req@ffff995b84642a00 x1855007616490112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076036.073666:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616490112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076036.073686:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076036.073692:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076036.073697:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616490112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076036.073706:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616490240:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076036.073727:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076036.073739:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076036/real 1769076036] req@ffff995b84643800 x1855007616490240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076036.073753:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616490240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076036.073764:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076036.073767:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076036.073772:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616490240:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076036.073795:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616490368:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076036.073814:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076036.073825:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076036/real 1769076036] req@ffff995b84643b80 x1855007616490368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076036.073840:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616490368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076036.073851:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076036.073854:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076036.073857:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616490368:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076036.073864:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616490496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076036.073881:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076036.073892:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076036/real 1769076036] req@ffff995b84641880 x1855007616490496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076036.073906:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616490496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076036.073916:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076036.073920:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076036.073923:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616490496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076036.073929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616490624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076036.073946:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076036.073958:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076036/real 1769076036] req@ffff995b84641500 x1855007616490624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076036.073972:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616490624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076036.073982:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076036.073986:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076036.073989:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616490624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076036.776596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076036.776724:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076037.096616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076037.096636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076037.096654:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076037.096665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076037.800587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076037.800666:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076038.120604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076038.120612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076038.120617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076038.120650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076038.824577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076038.824659:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076039.144572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076039.144603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076039.144636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076039.144642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076039.848577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076039.848680:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076040.168570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076040.168606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076040.168612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076040.168634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076040.872545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076040.872774:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076041.192721:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076041.192723:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076041.192723:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076041.192724:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076041.192818:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076041.192827:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076041.192835:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076041.192853:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3026 00000100:00080000:0.0:1769076041.192867:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076041.192875:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076041.192924:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616490752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076041.192966:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076041.192974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076041.192978:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076041.192987:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3026 00000100:00080000:0.0:1769076041.192996:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076041.193005:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076041.193021:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076041.193024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616490752:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076041.193060:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616490880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076041.193089:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076041.193095:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076041.193099:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076041.193105:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3026 00000100:00080000:0.0:1769076041.193112:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076041.193122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076041.193123:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076041.193154:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076041.193183:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616491008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076041.193189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076041/real 1769076041] req@ffff995b84643480 x1855007616490752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076041.193245:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076041.193255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616490752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076041.193258:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076041.193268:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076041.193282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076041.193285:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3026 00000100:00080000:0.0:1769076041.193301:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076041.193306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076041.193314:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076041.193328:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076041.193328:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616490752:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076041.193372:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616491136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076041.193420:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076041.193427:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076041.193431:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076041.193440:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3026 00000100:00080000:0.0:1769076041.193447:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076041.193501:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076041.193513:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076041.193517:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616490880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076041.193552:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616491264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076041.193574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769076041.193576:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076041.193590:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076041.193595:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076041.193604:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076041/real 1769076041] req@ffff995b84643100 x1855007616490880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076041.193610:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3026 00000100:00080000:0.0:1769076041.193629:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076041.193639:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616490880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076041.193646:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076041.193667:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769076041.193671:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076041.193676:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076041.193684:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616490880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076041.193695:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076041.193704:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616491392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076041.193722:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076041.193731:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1769076041.193733:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076041.193736:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076041.193746:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3026 00000100:00080000:0.0:1769076041.193754:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076041.193757:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076041/real 1769076041] req@ffff995b84642d80 x1855007616491008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076041.193766:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076041.193784:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076041.193794:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616491008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076041.193807:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616491520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076041.193814:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076041.193832:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076041.193846:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076041.193893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491136:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076041.193920:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076041.193933:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076041/real 1769076041] req@ffff995b84641500 x1855007616491136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076041.193948:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616491136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076041.193959:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076041.193962:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076041.193966:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491136:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076041.193973:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491264:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076041.193990:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076041.194001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076041/real 1769076041] req@ffff995b84641880 x1855007616491264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076041.194015:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616491264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076041.194025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076041.194029:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076041.194032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076041.194038:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616491392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076041.194056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076041.194066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076041/real 1769076041] req@ffff995b84643b80 x1855007616491392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076041.194080:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616491392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076041.194091:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076041.194095:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076041.194099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616491392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076041.194105:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616491520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076041.194122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076041.194132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076041/real 1769076041] req@ffff995b84643800 x1855007616491520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076041.194145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616491520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076041.194155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076041.194159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076041.194162:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616491520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076041.896616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076041.896710:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076042.216623:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076042.216665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076042.216680:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076042.216794:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076042.920547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076042.920646:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076043.240601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076043.240652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076043.240653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076043.240672:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076043.944590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076043.944684:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076044.264591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076044.264602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076044.264603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076044.264672:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076044.968586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076044.968694:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076045.288604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076045.288606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076045.288618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076045.288618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076045.992568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076045.992677:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076046.312609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076046.312630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076046.312633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076046.312644:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076046.312653:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076046.312667:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3032 00000100:00100000:3.0:1769076046.312673:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076046.312683:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076046.312700:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076046.312700:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076046.312757:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616491648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076046.312796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076046.312813:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076046.312817:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076046.312825:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3032 00000100:00080000:0.0:1769076046.312836:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076046.312844:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076046.312857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616491648:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076046.312858:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076046.312886:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616491776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076046.312902:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076046.312908:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076046.312913:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076046.312922:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3032 00000100:00080000:0.0:1769076046.312940:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076046.312948:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076046.312958:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076046.312972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076046.312987:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616491904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076046.313010:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076046.313018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076046.313031:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076046.313039:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3032 00000100:00080000:0.0:1769076046.313049:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076046.313057:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076046/real 1769076046] req@ffff995b84642a00 x1855007616491648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076046.313063:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076046.313079:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076046.313110:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616492032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076046.313119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616491648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076046.313133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076046.313147:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076046.313153:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076046.313166:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3032 00000100:00080000:2.0:1769076046.313167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076046.313171:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076046.313182:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076046.313189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769076046.313191:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076046.313209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616491648:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076046.313215:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616492160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076046.313232:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076046.313242:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076046.313245:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076046.313252:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3032 00000100:00080000:0.0:1769076046.313257:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076046.313262:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076046.313267:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076046.313280:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076046.313286:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616492288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076046.313362:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076046.313367:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076046.313373:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076046.313375:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076046.313390:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3032 00000100:00080000:0.0:1769076046.313399:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076046.313410:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076046.313411:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076046/real 1769076046] req@ffff995b84640e00 x1855007616491776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076046.313419:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076046.313446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616491776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076046.313507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769076046.313509:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616492416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076046.313520:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076046.313533:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076046.313553:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076046.313593:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076046.313611:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076046/real 1769076046] req@ffff995b84641180 x1855007616491904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076046.313650:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616491904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076046.313670:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076046.313676:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076046.313684:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616491904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076046.313697:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492032:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076046.313739:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076046.313757:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076046/real 1769076046] req@ffff995b84640000 x1855007616492032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076046.313773:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616492032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076046.313785:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076046.313789:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076046.313794:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492032:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076046.313801:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076046.313823:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076046.313835:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076046/real 1769076046] req@ffff995b84641c00 x1855007616492160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076046.313849:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616492160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076046.313868:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076046.313872:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076046.313876:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076046.313900:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616492288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076046.313930:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076046.313942:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076046/real 1769076046] req@ffff995b84642680 x1855007616492288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076046.313955:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616492288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076046.313966:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076046.313969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076046.313973:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616492288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076046.313984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616492416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076046.314018:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076046.314044:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076046/real 1769076046] req@ffff995b84640700 x1855007616492416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076046.314070:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616492416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076046.314088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076046.314094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076046.314101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616492416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076047.016590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076047.016739:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076047.336670:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076047.336687:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076047.336714:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076047.336784:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076048.040616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076048.040707:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076048.360571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076048.360610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076048.360673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076048.360686:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076049.064575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076049.064698:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076049.384561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076049.384577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076049.384600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076049.384658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076050.088556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076050.088703:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076050.408627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076050.408639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076050.408655:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076050.408691:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076051.112553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076051.112750:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076051.432571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076051.432578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076051.432582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076051.432604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076051.432609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076051.432614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076051.432618:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076051.432626:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3037 00000100:00080000:1.0:1769076051.432636:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076051.432642:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076051.432674:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616492544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076051.432695:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076051.432698:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076051.432700:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076051.432703:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3037 00000100:00080000:1.0:1769076051.432706:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076051.432709:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076051.432717:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076051.432727:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616492544:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076051.432729:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616492672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076051.432739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076051.432742:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076051.432744:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076051.432747:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3037 00000100:00080000:1.0:1769076051.432749:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076051.432751:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076051.432754:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076051.432764:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616492800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076051.432770:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076051.432772:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076051.432773:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076051.432775:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3037 00000100:00080000:1.0:1769076051.432779:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076051.432781:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076051.432785:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076051.432796:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616492928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076051.432796:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076051.432806:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076051.432809:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076051.432811:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076051.432813:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3037 00000100:00080000:1.0:1769076051.432816:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076051.432818:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076051.432822:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076051.432829:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616493056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076051.432836:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076051/real 1769076051] req@ffff995b868c7100 x1855007616492544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076051.432871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076051.432874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076051.432875:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076051.432877:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616492544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076051.432881:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3037 00000100:00080000:1.0:1769076051.432884:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076051.432887:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076051.432890:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076051.432902:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076051.432903:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616493184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076051.432911:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076051.432918:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076051.432919:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616492544:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076051.432924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076051.432927:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076051.432931:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3037 00000100:00080000:1.0:1769076051.432933:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076051.432935:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076051.432938:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076051.432945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076051.432947:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616493312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076051.432971:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076051.432981:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076051/real 1769076051] req@ffff995b868c6300 x1855007616492672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076051.432995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616492672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076051.433001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076051.433003:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076051.433005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076051.433009:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076051.433055:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076051.433067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076051/real 1769076051] req@ffff995b868c6d80 x1855007616492800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076051.433079:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616492800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076051.433089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076051.433092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076051.433096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076051.433102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492928:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076051.433119:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076051.433128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076051/real 1769076051] req@ffff995b868c5c00 x1855007616492928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076051.433139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616492928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076051.433154:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076051.433157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076051.433160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616492928:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076051.433165:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493056:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076051.433179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076051.433190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076051/real 1769076051] req@ffff995b868c4a80 x1855007616493056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076051.433260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616493056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076051.433270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076051.433274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076051.433278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076051.433285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616493184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076051.433307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076051.433316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076051/real 1769076051] req@ffff995b868c7800 x1855007616493184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076051.433324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616493184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076051.433330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076051.433332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076051.433335:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616493184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076051.433349:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616493312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076051.433361:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076051.433367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076051/real 1769076051] req@ffff995b868c5500 x1855007616493312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076051.433375:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616493312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076051.433380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076051.433382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076051.433384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616493312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076052.136569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076052.136698:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076052.456552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076052.456613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076052.456638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076052.456646:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076053.160543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076053.160680:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076053.480611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076053.480621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076053.480702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076053.480761:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076054.184584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076054.184753:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076054.504612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076054.504613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076054.504615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076054.504623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076055.208580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076055.208711:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076055.528547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076055.528589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076055.528624:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076055.528629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076056.232592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076056.232691:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076056.552591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076056.552628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076056.552638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076056.552673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076056.552704:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076056.552713:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076056.552735:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076056.552755:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3042 00000100:00080000:2.0:1769076056.552773:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076056.552787:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076056.552845:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616493440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076056.552894:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076056.552903:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076056.552907:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076056.552915:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3042 00000100:00080000:2.0:1769076056.552925:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076056.552935:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076056.552952:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076056.552964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616493440:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076056.552989:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616493568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076056.553013:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076056.553018:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076056.553021:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076056.553027:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3042 00000100:00080000:2.0:1769076056.553033:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076056.553038:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076056.553047:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076056.553060:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076056.553071:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616493696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076056.553095:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076056.553099:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076056.553103:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076056.553117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076056/real 1769076056] req@ffff995b85acaa00 x1855007616493440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076056.553164:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3042 00000100:00080000:2.0:1769076056.553175:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076056.553179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616493440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076056.553191:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076056.553205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076056.553210:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769076056.553218:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076056.553227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616493440:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076056.553249:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616493824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076056.553271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493568:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076056.553279:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076056.553288:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076056.553291:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076056.553299:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3042 00000100:00080000:2.0:1769076056.553306:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076056.553312:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076056.553320:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076056.553339:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076056.553343:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616493952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076056.553360:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076056.553378:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769076056.553382:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076056/real 1769076056] req@ffff995b85ac9880 x1855007616493568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076056.553383:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076056.553400:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3042 00000100:00080000:2.0:1769076056.553410:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076056.553422:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076056.553429:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076056.553436:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616493568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076056.553451:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616494080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076056.553500:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076056.553506:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076056.553507:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076056.553515:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076056.553523:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076056.553526:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3042 00000100:00080000:2.0:1769076056.553536:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076056.553541:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493568:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076056.553548:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076056.553563:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076056.553573:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076056.553585:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616494208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076056.553619:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076056.553648:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076056/real 1769076056] req@ffff995b85aca300 x1855007616493696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076056.553674:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616493696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076056.553693:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076056.553700:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076056.553708:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076056.553741:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493824:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076056.553776:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076056.553793:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076056/real 1769076056] req@ffff995b85acb800 x1855007616493824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076056.553813:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616493824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076056.553831:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076056.553838:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076056.553844:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493824:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076056.553857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493952:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076056.553895:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076056.553918:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076056/real 1769076056] req@ffff995b85ac9c00 x1855007616493952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076056.553945:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616493952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076056.553958:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076056.553962:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076056.553967:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616493952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076056.553975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616494080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076056.554005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076056.554019:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076056/real 1769076056] req@ffff995b85acb100 x1855007616494080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076056.554034:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616494080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076056.554045:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076056.554049:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076056.554053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616494080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076056.554060:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616494208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076056.554077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076056.554088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076056/real 1769076056] req@ffff995b85ac8380 x1855007616494208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076056.554102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616494208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076056.554145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076056.554151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076056.554156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616494208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076057.256582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076057.256855:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076057.576597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076057.576607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076057.576618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076057.576637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076058.280574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076058.280680:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076058.600564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076058.600566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076058.600639:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076058.600738:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076059.304568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076059.304767:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076059.624557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076059.624588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076059.624597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076059.624605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076060.328499:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076060.328678:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076060.648575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076060.648604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076060.648645:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076060.648680:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076061.352560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076061.352610:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076061.672559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076061.672620:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076061.672630:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769076061.672634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076061.672636:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076061.672652:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3047 00000100:00080000:3.0:1769076061.672681:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076061.672685:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076061.672694:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076061.672747:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076061.672770:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616494336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076061.672805:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076061.672810:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076061.672813:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076061.672837:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3047 00000100:00080000:3.0:1769076061.672846:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076061.672853:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076061.672867:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076061.672910:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616494464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076061.672927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616494336:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076061.672930:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076061.672938:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076061.672943:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076061.672950:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3047 00000100:00080000:3.0:1769076061.672954:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076061.672959:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076061.672964:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076061.672980:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616494592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076061.672990:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076061.672994:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076061.672996:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076061.673000:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3047 00000800:00000100:0.0:1769076061.673026:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076061.673121:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076061.673138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076061/real 1769076061] req@ffff995b8455b480 x1855007616494336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076061.673176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616494336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076061.673194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076061.673200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076061.673207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616494336:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076061.673241:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616494464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076061.673281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076061.673318:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076061.673324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076061/real 1769076061] req@ffff995b84559880 x1855007616494464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076061.673327:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076061.673350:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616494720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076061.673356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616494464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076061.673366:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076061.673376:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076061.673385:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076061.673409:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076061.673416:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616494464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076061.673431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616494592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076061.673468:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076061.673511:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076061.673519:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3047 00000100:00080000:3.0:1769076061.673528:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076061.673533:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076061/real 1769076061] req@ffff995b8455ad80 x1855007616494592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076061.673540:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076061.673556:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076061.673564:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616494592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076061.673578:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616494848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076061.673579:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076061.673592:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076061.673605:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616494592:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076061.673612:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076061.673618:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076061.673620:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076061.673627:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3047 00000100:00080000:3.0:1769076061.673632:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076061.673637:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616494720:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076061.673642:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076061.673668:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076061.673689:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616494976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076061.673698:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076061.673714:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076061.673721:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076061.673726:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076061/real 1769076061] req@ffff995b8455a300 x1855007616494720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076061.673727:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076061.673740:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3047 00000100:00080000:3.0:1769076061.673771:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076061.673781:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076061.673787:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616494720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076061.673793:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076061.673814:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076061.673821:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076061.673828:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616495104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076061.673835:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616494720:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076061.673869:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616494848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076061.673905:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076061.673919:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076061/real 1769076061] req@ffff995b8455b100 x1855007616494848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076061.673951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616494848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076061.673963:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076061.673967:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076061.673971:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616494848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076061.673989:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616494976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076061.674010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076061.674023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076061/real 1769076061] req@ffff995b84559c00 x1855007616494976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076061.674081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616494976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076061.674090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076061.674094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076061.674097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616494976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076061.674104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616495104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076061.674125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076061.674137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076061/real 1769076061] req@ffff995b84558700 x1855007616495104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076061.674149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616495104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076061.674157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076061.674160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076061.674164:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616495104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076062.376559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076062.376640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076062.696588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076062.696635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076062.696639:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076062.696650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076063.400594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076063.400675:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076063.720539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076063.720548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076063.720606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076063.720656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076064.424612:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076064.424714:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076064.744582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076064.744747:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076064.744759:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076064.744764:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076065.448585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076065.448766:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076065.768614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076065.768722:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076065.768724:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076065.768734:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076066.472564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076066.472788:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076066.792658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076066.792717:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076066.792726:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076066.792730:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076066.792732:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076066.792745:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076066.792766:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076066.792767:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3052 00000100:00080000:2.0:1769076066.792791:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076066.792803:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076066.792860:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616495232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076066.792902:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076066.792908:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076066.792911:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076066.792916:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3052 00000100:00080000:2.0:1769076066.792923:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076066.792930:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076066.792942:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076066.792991:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616495360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076066.793008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616495232:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076066.793010:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076066.793027:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076066.793031:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076066.793038:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3052 00000100:00080000:2.0:1769076066.793043:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076066.793048:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076066.793055:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076066.793073:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616495488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076066.793085:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076066.793090:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076066.793091:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076066.793095:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3052 00000100:00080000:2.0:1769076066.793101:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076066.793105:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076066.793113:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076066.793134:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616495616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076066.793152:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076066.793159:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076066.793163:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076066.793172:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3052 00000100:00100000:0.0:1769076066.793181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616495360:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076066.793184:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076066.793202:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076066.793215:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076066.793231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616495488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076066.793249:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616495744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076066.793269:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076066.793274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616495616:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769076066.793276:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076066.793286:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076066.793297:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3052 00000100:00080000:2.0:1769076066.793306:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076066.793313:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076066.793322:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076066.793328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616495744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076066.793350:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616495872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076066.793368:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076066.793373:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076066.793376:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076066.793381:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616495872:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076066.793388:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3052 00000100:00080000:2.0:1769076066.793399:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076066.793404:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076066.793412:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076066.793434:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616496000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076066.793531:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616496000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076067.496571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076067.496688:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076067.816586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076067.816607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076067.816716:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076067.816735:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076068.520591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076068.520731:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076068.840655:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076068.840705:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076068.840741:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076068.840786:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076069.544633:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076069.544746:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076069.864543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076069.864545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076069.864559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076069.864692:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769076069.864707:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769076069.864752:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076069.864773:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076069.864814:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076069.864827:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076069.864836:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076069.864849:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076069.864858:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076066/real 1769076069] req@ffff995b85acbb80 x1855007616495360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1769076069.864859:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076069.864874:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076069.864893:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616495360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076069.864923:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076069.864931:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076069.864937:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616495360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076069.864946:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076066/real 1769076069] req@ffff995b85ac8380 x1855007616495488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076069.864959:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616495488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076069.864970:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076069.864974:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076069.864978:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616495488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076069.864985:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076066/real 1769076069] req@ffff995b85acb100 x1855007616495616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076069.864997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616495616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076069.865006:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076069.865011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076069.865014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616495616:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076069.865021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076066/real 1769076069] req@ffff995b85ac8a80 x1855007616495232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076069.865035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616495232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076069.865045:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076069.865048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076069.865053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616495232:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076069.865101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076066/real 1769076069] req@ffff995b85aca300 x1855007616496000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076069.865118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616496000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076069.865127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076069.865132:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076069.865135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616496000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076069.865143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076066/real 1769076069] req@ffff995b85acb800 x1855007616495872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076069.865156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616495872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076069.865166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076069.865170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076069.865174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616495872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076069.865181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076066/real 1769076069] req@ffff995b85ac9c00 x1855007616495744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076069.865194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616495744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076069.865204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076069.865208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076069.865212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616495744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076070.568601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076070.568712:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076070.888645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076070.888654:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076070.888706:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076070.888708:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076071.592580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076071.592717:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076071.912664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076071.912695:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076071.912774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076071.912776:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076071.912782:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076071.912790:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076071.912797:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076071.912810:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3057 00000100:00080000:0.0:1769076071.912824:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076071.912834:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076071.912931:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616496128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076071.912971:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076071.912977:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076071.912980:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076071.912985:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3057 00000100:00080000:0.0:1769076071.912991:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076071.912996:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076071.913007:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076071.913024:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616496256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076071.913039:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076071.913044:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076071.913049:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076071.913053:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616496128:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076071.913057:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3057 00000100:00080000:0.0:1769076071.913067:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076071.913075:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076071.913081:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076071.913096:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616496384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076071.913109:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076071.913115:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076071.913119:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076071.913127:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3057 00000100:00080000:0.0:1769076071.913134:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076071.913143:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076071.913154:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076071.913173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076071.913185:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616496512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076071.913220:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076071.913227:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076071.913231:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076071.913240:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3057 00000100:00080000:0.0:1769076071.913251:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076071.913260:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076071.913270:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076071.913278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076071/real 1769076071] req@ffff995b84640a80 x1855007616496128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076071.913294:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616496640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076071.913322:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076071.913332:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616496128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076071.913334:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076071.913342:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076071.913359:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3057 00000100:00080000:0.0:1769076071.913375:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076071.913380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076071.913388:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076071.913399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769076071.913404:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076071.913413:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616496128:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076071.913435:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616496768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076071.913512:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616496256:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076071.913537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076071.913546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076071.913551:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076071.913562:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3057 00000100:00080000:0.0:1769076071.913572:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076071.913575:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076071.913586:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076071.913599:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076071.913619:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076071/real 1769076071] req@ffff995b84641f80 x1855007616496256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076071.913629:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616496896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076071.913674:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616496256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076071.913698:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076071.913706:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076071.913714:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616496256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076071.913731:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616496384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076071.913775:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076071.913800:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076071/real 1769076071] req@ffff995b84640380 x1855007616496384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076071.913824:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616496384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076071.913844:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076071.913851:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076071.913857:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616496384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076071.913899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616496512:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076071.913944:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076071.913969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076071/real 1769076071] req@ffff995b84640700 x1855007616496512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076071.913995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616496512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076071.914045:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076071.914053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076071.914060:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616496512:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076071.914074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616496640:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076071.914117:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076071.914141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076071/real 1769076071] req@ffff995b84642680 x1855007616496640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076071.914164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616496640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076071.914183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076071.914190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076071.914198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616496640:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076071.914235:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616496768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076071.914276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076071.914301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076071/real 1769076071] req@ffff995b84641c00 x1855007616496768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076071.914325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616496768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076071.914345:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076071.914353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076071.914363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616496768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076071.914378:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616496896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076071.914417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076071.914439:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076071/real 1769076071] req@ffff995b84640000 x1855007616496896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076071.914519:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616496896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076071.914539:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076071.914547:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076071.914554:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616496896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076072.616551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076072.616692:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076072.936640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076072.936643:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076072.936741:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076072.936796:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076073.640604:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076073.640766:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076073.960644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076073.960654:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076073.960685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076073.960802:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076074.664600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076074.664762:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076074.984679:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076074.984680:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076074.984681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076074.984763:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076075.688621:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076075.688734:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076076.008571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076076.008643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076076.008650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076076.008685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076076.712570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076076.712710:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076077.032611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076077.032634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076077.032650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076077.032692:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076077.032720:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076077.032730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076077.032736:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076077.032748:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3062 00000100:00080000:3.0:1769076077.032763:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076077.032774:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076077.032847:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616497024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076077.032883:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076077.032888:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076077.032892:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076077.032897:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3062 00000100:00080000:3.0:1769076077.032904:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076077.032911:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076077.032923:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076077.032944:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616497024:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076077.032945:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616497152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076077.032971:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076077.032975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076077.032977:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076077.032982:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3062 00000100:00080000:3.0:1769076077.032986:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076077.032990:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076077.032998:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076077.033014:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616497280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076077.033029:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076077.033036:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076077.033040:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076077.033049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076077.033054:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3062 00000100:00080000:3.0:1769076077.033068:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076077.033079:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076077.033090:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076077.033108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076077/real 1769076077] req@ffff995b84558000 x1855007616497024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076077.033122:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616497408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076077.033154:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616497024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076077.033177:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076077.033183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076077.033185:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076077.033193:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076077.033200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076077.033208:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3062 00000100:00100000:2.0:1769076077.033216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616497024:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076077.033220:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076077.033232:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076077.033240:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076077.033255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616497152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076077.033265:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616497536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076077.033284:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076077.033289:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076077.033292:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076077.033299:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076077.033304:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3062 00000100:00080000:3.0:1769076077.033314:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076077.033320:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076077.033327:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076077.033329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076077/real 1769076077] req@ffff995b8455bb80 x1855007616497152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076077.033350:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616497664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076077.033356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616497152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076077.033380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076077.033388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076077.033400:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076077.033402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616497152:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076077.033412:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076077.033417:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076077.033421:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616497280:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076077.033429:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3062 00000100:00080000:3.0:1769076077.033437:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076077.033442:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076077.033490:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076077.033491:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076077.033516:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076077/real 1769076077] req@ffff995b8455aa00 x1855007616497280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076077.033522:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616497792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076077.033561:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616497280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076077.033586:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076077.033593:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076077.033598:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616497280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076077.033608:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616497408:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076077.033635:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076077.033647:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076077/real 1769076077] req@ffff995b84559500 x1855007616497408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076077.033661:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616497408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076077.033672:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076077.033675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076077.033679:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616497408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076077.033697:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616497536:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076077.033715:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076077.033726:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076077/real 1769076077] req@ffff995b84558a80 x1855007616497536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076077.033742:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616497536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076077.033753:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076077.033758:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076077.033765:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616497536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076077.033776:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616497664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076077.033842:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076077.033860:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076077/real 1769076077] req@ffff995b8455a680 x1855007616497664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076077.033875:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616497664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076077.033885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076077.033889:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076077.033893:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616497664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076077.033900:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616497792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076077.033918:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076077.033929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076077/real 1769076077] req@ffff995b84558000 x1855007616497792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076077.033943:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616497792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076077.033953:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076077.033956:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076077.033960:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616497792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076077.736603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076077.736679:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076078.056592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076078.056592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076078.056652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076078.056653:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076078.760633:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076078.760843:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076079.080596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076079.080648:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076079.080726:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076079.081263:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076079.784591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076079.784662:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076080.104594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076080.104600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076080.104641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076080.104646:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076080.808544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076080.808683:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076081.128538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076081.128562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076081.128571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076081.128585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076081.832586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076081.832735:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076082.152605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076082.152721:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076082.152807:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076082.152875:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076082.152889:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076082.152896:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076082.152906:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076082.152912:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3067 00000100:00080000:1.0:1769076082.152930:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076082.152943:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076082.153000:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616497920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076082.153043:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076082.153051:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076082.153054:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076082.153062:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3067 00000100:00080000:1.0:1769076082.153069:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076082.153076:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076082.153089:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076082.153096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616497920:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076082.153116:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616498048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076082.153133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076082.153138:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076082.153141:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076082.153147:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3067 00000100:00080000:1.0:1769076082.153153:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076082.153158:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076082.153163:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076082.153182:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616498176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076082.153192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076082.153209:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076082.153215:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076082.153219:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076082.153225:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3067 00000100:00080000:1.0:1769076082.153230:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076082.153234:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076082.153241:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076082.153257:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616498304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076082.153271:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076082.153279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076082/real 1769076082] req@ffff995b868c7480 x1855007616497920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076082.153280:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076082.153286:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076082.153297:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3067 00000100:00080000:1.0:1769076082.153305:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076082.153315:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076082.153326:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076082.153344:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616498432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076082.153352:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616497920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076082.153365:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076082.153381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076082.153387:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076082.153396:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076082.153398:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3067 00000100:00080000:1.0:1769076082.153409:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076082.153413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076082.153420:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076082.153429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616497920:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076082.153433:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076082.153487:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616498560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076082.153506:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498048:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076082.153509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076082.153522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076082.153527:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076082.153534:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3067 00000100:00080000:1.0:1769076082.153541:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076082.153546:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076082.153553:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076082.153562:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076082.153576:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616498688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076082.153591:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076082/real 1769076082] req@ffff995b868c4700 x1855007616498048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076082.153614:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616498048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076082.153627:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076082.153633:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076082.153638:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076082.153647:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076082.153669:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076082.153688:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076082/real 1769076082] req@ffff995b868c6a00 x1855007616498176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076082.153744:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616498176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076082.153764:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076082.153771:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076082.153779:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076082.153793:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498304:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076082.153834:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076082.153852:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076082/real 1769076082] req@ffff995b868c5500 x1855007616498304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076082.153868:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616498304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076082.153879:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076082.153884:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076082.153888:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498304:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076082.153896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498432:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076082.153917:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076082.153929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076082/real 1769076082] req@ffff995b868c7800 x1855007616498432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076082.153942:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616498432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076082.153953:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076082.153957:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076082.153960:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498432:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076082.153984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616498560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076082.154003:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076082.154014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076082/real 1769076082] req@ffff995b868c4a80 x1855007616498560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076082.154028:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616498560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076082.154038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076082.154042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076082.154045:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616498560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076082.154052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616498688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076082.154069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076082.154080:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076082/real 1769076082] req@ffff995b868c5c00 x1855007616498688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076082.154103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616498688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076082.154113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076082.154116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076082.154120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616498688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076082.856601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076082.856736:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076083.176549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076083.176571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076083.176576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076083.176608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076083.880578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076083.880684:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076084.200559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076084.200566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076084.200590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076084.200633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076084.904588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076084.904676:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076085.224579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076085.224604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076085.224605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076085.224674:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076085.928580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076085.928690:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076086.248601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076086.248606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076086.248616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076086.248712:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076086.952662:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076086.952792:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076087.272644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076087.272710:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076087.272710:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076087.272746:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076087.272778:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076087.272785:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076087.272790:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076087.272802:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3072 00000100:00080000:2.0:1769076087.272815:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076087.272822:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076087.272860:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616498816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076087.272890:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076087.272894:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076087.272896:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076087.272900:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3072 00000100:00080000:2.0:1769076087.272905:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076087.272910:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076087.272921:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076087.272935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616498816:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076087.272937:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616498944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076087.272965:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076087.272969:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076087.272972:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076087.272976:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3072 00000100:00080000:2.0:1769076087.272979:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076087.272983:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076087.272988:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076087.273000:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616499072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076087.273010:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076087.273014:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1769076087.273019:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076087.273020:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076087.273030:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3072 00000100:00080000:2.0:1769076087.273035:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076087.273039:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076087.273045:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076087.273058:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616499200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076087.273074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076087/real 1769076087] req@ffff995b85ac9880 x1855007616498816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076087.273077:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076087.273095:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076087.273100:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076087.273105:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3072 00000100:00080000:2.0:1769076087.273109:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076087.273117:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076087.273119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616498816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076087.273123:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076087.273141:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616499328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076087.273147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076087.273153:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076087.273160:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076087.273161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076087.273163:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076087.273173:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3072 00000100:00100000:0.0:1769076087.273178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616498816:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076087.273179:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076087.273190:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076087.273204:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076087.273216:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616499456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076087.273228:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076087.273231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498944:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076087.273233:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076087.273240:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076087.273247:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3072 00000100:00080000:2.0:1769076087.273252:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076087.273256:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076087.273261:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076087.273272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076087.273277:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616499584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076087.273306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076087/real 1769076087] req@ffff995b85acaa00 x1855007616498944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076087.273327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616498944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076087.273340:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076087.273346:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076087.273351:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616498944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076087.273361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076087.273393:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076087.273410:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076087/real 1769076087] req@ffff995b85ac9c00 x1855007616499072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076087.273423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616499072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076087.273433:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076087.273436:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076087.273439:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076087.273446:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499200:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076087.273496:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076087.273508:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076087/real 1769076087] req@ffff995b85acb800 x1855007616499200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076087.273519:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616499200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076087.273529:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076087.273532:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076087.273535:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499200:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076087.273540:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499328:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076087.273555:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076087.273564:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076087/real 1769076087] req@ffff995b85aca300 x1855007616499328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076087.273574:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616499328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076087.273582:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076087.273585:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076087.273587:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499328:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076087.273630:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616499456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076087.273653:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076087.273668:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076087/real 1769076087] req@ffff995b85ac8a80 x1855007616499456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076087.273680:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616499456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076087.273689:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076087.273691:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076087.273694:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616499456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076087.273700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616499584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076087.273714:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076087.273722:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076087/real 1769076087] req@ffff995b85acb100 x1855007616499584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076087.273733:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616499584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076087.273741:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076087.273744:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076087.273747:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616499584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076087.976562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076087.976692:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076088.296573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076088.296636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076088.296647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076088.296672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076089.000554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076089.000704:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076089.320549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076089.320649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076089.320687:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076089.320735:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076090.024555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076090.024718:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076090.344680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076090.344685:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076090.344727:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076090.344776:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076091.048632:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076091.048700:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076091.368603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076091.368614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076091.368663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076091.368703:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076092.072624:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076092.072705:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076092.392626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076092.392654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076092.392664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076092.392666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769076092.392671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076092.392677:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076092.392698:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3078 00000100:00080000:3.0:1769076092.392721:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076092.392735:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076092.392747:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076092.392810:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616499712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076092.392854:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076092.392861:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076092.392867:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076092.392875:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3078 00000100:00080000:3.0:1769076092.392883:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076092.392892:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076092.392908:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076092.392929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616499712:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076092.392945:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616499840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076092.392966:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076092.392971:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076092.392975:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076092.392982:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3078 00000100:00080000:3.0:1769076092.392987:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076092.392993:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076092.393003:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076092.393022:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616499968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076092.393023:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076092.393040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076092.393045:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076092.393048:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076092.393055:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3078 00000100:00080000:3.0:1769076092.393060:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076092.393067:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076092.393076:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076092.393080:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076092/real 1769076092] req@ffff995b84558700 x1855007616499712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076092.393094:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616500096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076092.393111:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076092.393117:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076092.393121:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076092.393128:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3078 00000100:00100000:0.0:1769076092.393132:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616499712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076092.393135:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076092.393141:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076092.393150:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769076092.393151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076092.393160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076092.393168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616499712:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076092.393169:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616500224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076092.393186:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076092.393191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076092.393194:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076092.393204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499840:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076092.393216:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3078 00000100:00080000:3.0:1769076092.393222:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076092.393228:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076092.393238:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076092.393246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076092.393257:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616500352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076092.393270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076092/real 1769076092] req@ffff995b84559c00 x1855007616499840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076092.393277:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076092.393284:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076092.393287:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076092.393295:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3078 00000100:00100000:0.0:1769076092.393296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616499840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076092.393302:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076092.393309:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076092.393313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769076092.393317:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076092.393319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076092.393326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076092.393334:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616500480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076092.393339:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076092.393372:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076092.393394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076092/real 1769076092] req@ffff995b8455b100 x1855007616499968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076092.393420:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616499968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076092.393447:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076092.393451:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076092.393487:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616499968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076092.393496:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500096:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076092.393553:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076092.393569:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076092/real 1769076092] req@ffff995b8455a300 x1855007616500096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076092.393584:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616500096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076092.393596:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076092.393600:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076092.393604:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500096:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076092.393611:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500224:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076092.393631:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076092.393642:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076092/real 1769076092] req@ffff995b8455ad80 x1855007616500224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076092.393656:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616500224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076092.393667:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076092.393670:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076092.393674:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500224:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076092.393697:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616500352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076092.393715:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076092.393727:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076092/real 1769076092] req@ffff995b84559880 x1855007616500352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076092.393741:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616500352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076092.393751:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076092.393754:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076092.393758:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616500352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076092.393764:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616500480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076092.393781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076092.393792:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076092/real 1769076092] req@ffff995b8455b480 x1855007616500480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076092.393806:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616500480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076092.393815:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076092.393819:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076092.393822:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616500480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076093.096641:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076093.096738:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076093.416674:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076093.416674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076093.416683:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076093.416724:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076094.120634:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076094.120741:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076094.440644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076094.440644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076094.440648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076094.440746:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076095.144592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076095.144630:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076095.464556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076095.464567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076095.464568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076095.464621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076096.168572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076096.168757:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076096.488682:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076096.488718:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076096.488774:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076096.488783:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076097.192576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076097.192630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076097.512658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076097.512670:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076097.512674:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076097.512682:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076097.512697:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076097.512707:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076097.512725:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3083 00000100:00080000:3.0:1769076097.512747:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076097.512763:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076097.512793:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076097.512843:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616500608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076097.512891:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076097.512899:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076097.512903:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076097.512912:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3083 00000100:00080000:3.0:1769076097.512919:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076097.512928:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076097.512947:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076097.512976:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616500736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076097.512997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076097.513004:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076097.513008:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076097.513016:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3083 00000100:00080000:3.0:1769076097.513024:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076097.513033:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076097.513044:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076097.513065:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616500864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076097.513078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616500608:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076097.513103:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076097.513125:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076097.513128:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076097.513135:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3083 00000100:00080000:3.0:1769076097.513139:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076097.513147:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076097.513157:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076097.513174:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616500992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076097.513186:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076097.513190:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076097.513193:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076097.513198:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3083 00000100:00080000:3.0:1769076097.513204:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076097.513210:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076097.513216:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076097.513225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076097.513234:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616501120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076097.513252:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076097.513257:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076097.513259:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076097.513265:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3083 00000100:00080000:3.0:1769076097.513270:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076097.513277:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076097.513288:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076097.513303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076097/real 1769076097] req@ffff995b8455b480 x1855007616500608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076097.513310:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616501248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076097.513414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076097.513419:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076097.513422:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076097.513524:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616500608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076097.513525:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3083 00000100:00080000:3.0:1769076097.513537:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076097.513554:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076097.513568:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076097.513571:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076097.513586:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076097.513597:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616500608:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076097.513603:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616501376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076097.513629:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076097.513675:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076097.513694:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076097/real 1769076097] req@ffff995b84559880 x1855007616500736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076097.513708:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616500736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076097.513718:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076097.513722:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076097.513726:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076097.513759:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076097.513788:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076097.513803:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076097/real 1769076097] req@ffff995b8455ad80 x1855007616500864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076097.513818:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616500864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076097.513835:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076097.513840:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076097.513844:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076097.513851:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500992:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076097.513897:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076097.513913:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076097/real 1769076097] req@ffff995b8455a300 x1855007616500992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076097.513927:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616500992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076097.513939:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076097.513945:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076097.513949:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616500992:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076097.513958:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616501120:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076097.513982:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076097.513996:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076097/real 1769076097] req@ffff995b8455b100 x1855007616501120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076097.514035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616501120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076097.514047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076097.514057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076097.514062:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616501120:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076097.514075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616501248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076097.514094:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076097.514120:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076097/real 1769076097] req@ffff995b84559c00 x1855007616501248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076097.514140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616501248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076097.514153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076097.514157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076097.514160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616501248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076097.514169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616501376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076097.514192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076097.514203:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076097/real 1769076097] req@ffff995b84558700 x1855007616501376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076097.514221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616501376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076097.514262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076097.514271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076097.514281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616501376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076098.216661:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076098.216700:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076098.536623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076098.536635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076098.536641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076098.536693:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076099.240597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076099.240725:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076099.560636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076099.560655:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076099.560667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076099.560735:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076100.264594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076100.264698:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076100.584638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076100.584645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076100.584653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076100.584702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076101.288564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076101.288653:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076101.608628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076101.608674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076101.608683:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076101.608695:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076102.312594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076102.312657:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076102.632540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076102.632594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076102.632601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076102.632619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076102.632627:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076102.632633:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076102.632645:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3088 00000100:00100000:3.0:1769076102.632653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076102.632661:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076102.632675:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076102.632730:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616501504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076102.632761:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076102.632765:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076102.632766:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076102.632770:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3088 00000100:00080000:2.0:1769076102.632772:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076102.632775:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076102.632786:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076102.632797:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616501632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076102.632805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076102.632808:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076102.632810:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076102.632813:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3088 00000100:00080000:2.0:1769076102.632818:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076102.632822:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076102.632826:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076102.632841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616501504:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076102.632841:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616501760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076102.632873:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076102.632875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076102.632877:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076102.632881:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3088 00000100:00080000:2.0:1769076102.632884:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076102.632888:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076102.632895:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076102.632909:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616501888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076102.632915:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076102.632918:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076102.632919:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076102.632922:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3088 00000100:00080000:2.0:1769076102.632927:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076102.632930:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076102.632934:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076102.632942:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616502016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076102.632954:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076102.632962:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1769076102.632964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076102.632966:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076102.632974:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3088 00000100:00080000:2.0:1769076102.632979:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076102.632983:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076102.632988:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076102.633000:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616502144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076102.633007:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076102.633011:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076102.633013:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076102.633019:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3088 00000100:00080000:2.0:1769076102.633025:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076102.633026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076102/real 1769076102] req@ffff995b85ac8380 x1855007616501504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076102.633040:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076102.633054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076102.633078:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616502272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076102.633081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616501504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076102.633168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076102.633175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076102.633184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616501504:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076102.633196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616501632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076102.633222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076102.633236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076102/real 1769076102] req@ffff995b85acbb80 x1855007616501632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076102.633259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616501632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076102.633268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076102.633272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076102.633276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616501632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076102.633306:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616501760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076102.633325:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076102.633336:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076102/real 1769076102] req@ffff995b85acb100 x1855007616501760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076102.633388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616501760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076102.633401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076102.633405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076102.633409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616501760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076102.633417:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616501888:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076102.633439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076102.633451:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076102/real 1769076102] req@ffff995b85ac8a80 x1855007616501888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076102.633497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616501888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076102.633516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076102.633520:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076102.633523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616501888:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076102.633531:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502016:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076102.633553:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076102.633565:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076102/real 1769076102] req@ffff995b85aca300 x1855007616502016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076102.633577:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616502016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076102.633587:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076102.633591:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076102.633594:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502016:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076102.633600:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616502144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076102.633618:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076102.633628:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076102/real 1769076102] req@ffff995b85acb800 x1855007616502144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076102.633641:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616502144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076102.633650:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076102.633654:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076102.633658:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616502144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076102.633664:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616502272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076102.633680:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076102.633690:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076102/real 1769076102] req@ffff995b85ac9c00 x1855007616502272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076102.633702:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616502272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076102.633711:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076102.633715:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076102.633719:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616502272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076103.336590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076103.336731:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076103.656599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076103.656625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076103.656629:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076103.656658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076104.360618:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076104.360723:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076104.680638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076104.680652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076104.680703:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076104.680720:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076105.384565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076105.384711:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076105.704590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076105.704605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076105.704608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076105.704681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076106.408571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076106.408687:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076106.728602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076106.728604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076106.728651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076106.728703:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076107.432566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076107.432759:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076107.752566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076107.752609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076107.752650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076107.752665:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076107.752675:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076107.752680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076107.752681:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076107.752694:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3093 00000100:00080000:0.0:1769076107.752715:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076107.752726:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076107.752783:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616502400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076107.752840:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076107.752849:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076107.752853:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076107.752862:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3093 00000100:00080000:0.0:1769076107.752870:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076107.752878:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076107.752891:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076107.752896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616502400:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076107.752917:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616502528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076107.752939:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076107.752947:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076107.752951:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076107.752959:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3093 00000100:00080000:0.0:1769076107.752967:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076107.752976:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076107.752987:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076107.752995:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076107.753025:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616502656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076107.753049:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076107.753057:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076107/real 1769076107] req@ffff995b84641180 x1855007616502400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076107.753064:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076107.753074:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076107.753092:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3093 00000100:00100000:2.0:1769076107.753102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616502400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076107.753104:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076107.753123:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076107.753143:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769076107.753145:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769076107.753164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076107.753177:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616502784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076107.753182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616502400:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076107.753211:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076107.753216:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076107.753219:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076107.753226:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3093 00000100:00080000:0.0:1769076107.753231:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076107.753238:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076107.753247:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502528:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769076107.753252:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076107.753336:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076107.753345:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616502912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076107.753366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076107/real 1769076107] req@ffff995b84640e00 x1855007616502528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076107.753381:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076107.753393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616502528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076107.753395:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076107.753401:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076107.753418:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3093 00000100:00080000:2.0:1769076107.753425:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076107.753429:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076107.753438:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076107.753443:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076107.753486:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076107.753504:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502656:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769076107.753516:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076107.753543:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076107.753552:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616503040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076107.753569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076107.753571:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076107/real 1769076107] req@ffff995b84642a00 x1855007616502656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076107.753583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076107.753595:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076107.753605:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616502656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076107.753615:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3093 00000100:00080000:0.0:1769076107.753633:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076107.753634:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076107.753649:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076107.753656:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076107.753661:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502656:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769076107.753674:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076107.753676:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502784:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076107.753719:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616503168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076107.753724:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076107.753745:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076107/real 1769076107] req@ffff995b84643800 x1855007616502784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076107.753762:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616502784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076107.753773:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076107.753777:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076107.753781:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502784:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076107.753809:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502912:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076107.753830:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076107.753843:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076107/real 1769076107] req@ffff995b84643b80 x1855007616502912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076107.753858:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616502912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076107.753869:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076107.753872:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076107.753876:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616502912:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076107.753883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616503040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076107.753900:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076107.753912:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076107/real 1769076107] req@ffff995b84641880 x1855007616503040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076107.753926:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616503040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076107.753936:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076107.753939:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076107.753943:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616503040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076107.753949:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616503168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076107.753966:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076107.753977:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076107/real 1769076107] req@ffff995b84641500 x1855007616503168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076107.753991:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616503168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076107.754009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076107.754013:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076107.754017:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616503168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076108.456550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076108.456674:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076108.776614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076108.776628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076108.776663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076108.776681:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076109.480590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076109.480715:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076109.800639:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076109.800650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076109.800652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076109.800736:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076110.504588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076110.504712:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076110.824619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076110.824689:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076110.824733:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076110.824737:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076111.528571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076111.528771:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076111.848559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076111.848576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076111.848581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076111.848640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076112.552545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076112.552611:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076112.872585:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076112.872596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076112.872597:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769076112.872599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076112.872604:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076112.872616:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3098 00000100:00080000:2.0:1769076112.872628:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076112.872653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076112.872656:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076112.872734:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616503296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076112.872741:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076112.872769:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076112.872774:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076112.872777:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076112.872782:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3098 00000100:00080000:2.0:1769076112.872786:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076112.872792:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076112.872803:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076112.872815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616503296:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076112.872825:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616503424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076112.872845:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076112.872850:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076112.872854:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076112.872859:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3098 00000100:00080000:2.0:1769076112.872879:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076112.872892:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769076112.872901:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769076112.872903:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076112.872925:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616503552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076112.872940:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076112.872945:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076112.872949:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076112.872955:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3098 00000100:00100000:3.0:1769076112.872962:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076112/real 1769076112] req@ffff995b85acaa00 x1855007616503296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076112.872965:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076112.872978:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076112.872991:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076112.873007:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616503296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076112.873015:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616503680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076112.873037:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076112.873046:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076112.873048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076112.873059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616503296:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076112.873060:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076112.873068:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076112.873078:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3098 00000100:00080000:2.0:1769076112.873084:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076112.873092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616503424:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076112.873094:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076112.873107:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076112.873128:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616503808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076112.873135:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076112.873151:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076112.873155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076112/real 1769076112] req@ffff995b85ac9880 x1855007616503424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076112.873157:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076112.873162:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076112.873200:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3098 00000100:00100000:3.0:1769076112.873207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616503424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076112.873213:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076112.873222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076112.873225:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076112.873231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:2.0:1769076112.873238:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076112.873239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616503424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076112.873252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616503552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076112.873262:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616503936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076112.873274:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076112.873277:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076112.873279:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076112.873282:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3098 00000100:00080000:2.0:1769076112.873286:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076112.873291:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076112.873293:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076112.873301:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076112.873330:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076112/real 1769076112] req@ffff995b85ac9c00 x1855007616503552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076112.873338:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616504064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076112.873371:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616503552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076112.873386:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076112.873393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076112.873397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616503552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076112.873406:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616503680:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076112.873432:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076112.873444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076112/real 1769076112] req@ffff995b85acb800 x1855007616503680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076112.873483:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616503680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076112.873523:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076112.873530:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076112.873535:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616503680:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076112.873582:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616503808:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076112.873624:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076112.873642:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076112/real 1769076112] req@ffff995b85aca300 x1855007616503808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076112.873658:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616503808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076112.873670:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076112.873674:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076112.873678:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616503808:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076112.873702:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616503936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076112.873718:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076112.873727:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076112/real 1769076112] req@ffff995b85ac8a80 x1855007616503936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076112.873737:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616503936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076112.873751:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076112.873754:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076112.873757:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616503936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076112.873761:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616504064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076112.873774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076112.873781:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076112/real 1769076112] req@ffff995b85acb100 x1855007616504064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076112.873791:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616504064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076112.873798:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076112.873801:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076112.873803:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616504064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076113.576581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076113.576723:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076113.896583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076113.896630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076113.896648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076113.896686:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076114.600592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076114.600728:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076114.920604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076114.920633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076114.920656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076114.920672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076115.624581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076115.624721:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076115.944592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076115.944592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076115.944592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076115.944613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076116.648605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076116.648700:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076116.968599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076116.968651:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076116.968665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076116.968665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076117.672682:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076117.672733:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076117.992537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076117.992598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076117.992620:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076117.992632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076117.992633:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076117.992647:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076117.992664:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3103 00000100:00100000:3.0:1769076117.992673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076117.992696:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076117.992721:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076117.992771:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616504192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076117.992811:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076117.992817:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076117.992821:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076117.992829:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3103 00000100:00080000:2.0:1769076117.992836:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076117.992843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076117.992857:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076117.992882:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616504192:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076117.992898:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616504320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076117.992917:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076117.992923:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076117.992927:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076117.992933:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3103 00000100:00080000:2.0:1769076117.992958:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076117.992965:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076117.992974:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076117.992977:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076117.992995:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616504448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076117.993012:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076117.993018:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076117.993022:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076117.993028:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3103 00000100:00080000:2.0:1769076117.993034:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076117.993035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076117/real 1769076117] req@ffff995b85acbb80 x1855007616504192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076117.993041:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076117.993050:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076117.993067:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616504576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076117.993082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616504192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076117.993137:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076117.993144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076117.993150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616504192:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076117.993197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616504320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076117.993238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076117.993261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076117/real 1769076117] req@ffff995b85ac8380 x1855007616504320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076117.993284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616504320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076117.993299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076117.993305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076117.993311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616504320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076117.993321:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616504448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076117.993352:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076117.993384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076117/real 1769076117] req@ffff995b85acb100 x1855007616504448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076117.993400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616504448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076117.993437:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076117.993437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076117.993443:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076117.993443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076117.993446:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076117.993449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616504448:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076117.993487:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3103 00000100:00080000:2.0:1769076117.993494:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076117.993500:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076117.993503:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616504576:192.168.204.132@tcp:8:kworker.0 00000008:00100000:2.0:1769076117.993509:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076117.993528:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616504704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076117.993538:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076117.993563:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076117.993570:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076117.993574:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076117.993576:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076117/real 1769076117] req@ffff995b85ac8a80 x1855007616504576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076117.993581:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3103 00000100:00080000:2.0:1769076117.993588:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076117.993595:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076117.993600:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616504576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076117.993604:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076117.993620:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076117.993623:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616504832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076117.993627:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076117.993634:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616504576:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769076117.993641:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076117.993647:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076117.993650:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076117.993656:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616504704:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769076117.993657:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3103 00000100:00080000:2.0:1769076117.993663:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076117.993670:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076117.993697:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076117.993705:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076117.993710:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616504960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076117.993722:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076117/real 1769076117] req@ffff995b85aca300 x1855007616504704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076117.993740:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616504704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076117.993755:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076117.993761:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076117.993767:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616504704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076117.993777:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616504832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076117.993805:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076117.993819:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076117/real 1769076117] req@ffff995b85acb800 x1855007616504832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076117.993831:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616504832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076117.993840:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076117.993866:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076117.993872:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616504832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076117.993891:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616504960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076117.993913:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076117.993926:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076117/real 1769076117] req@ffff995b85ac9c00 x1855007616504960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076117.993940:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616504960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076117.993950:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076117.993953:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076117.993957:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616504960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076118.696584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076118.696675:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076119.016613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076119.016643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076119.016644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076119.016687:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076119.720561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076119.720713:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076120.040618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076120.040624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076120.040665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076120.040666:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076120.744554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076120.744743:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076121.064584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076121.064612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076121.064618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076121.064846:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076121.768592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076121.768732:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076122.088617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076122.088651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076122.088652:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076122.088697:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076122.792579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076122.792751:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076123.112591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076123.112597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076123.112609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076123.112647:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076123.112657:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076123.112664:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076123.112675:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076123.112681:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3108 00000100:00080000:0.0:1769076123.112701:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076123.112714:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076123.112770:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616505088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076123.112817:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076123.112822:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076123.112824:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076123.112830:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3108 00000100:00080000:0.0:1769076123.112837:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076123.112843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076123.112855:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076123.112875:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616505216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076123.112885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616505088:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076123.112897:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076123.112903:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076123.112905:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076123.112910:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3108 00000100:00080000:0.0:1769076123.112914:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076123.112919:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076123.112925:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076123.112941:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616505344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076123.112952:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076123.112956:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076123.112959:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076123.112963:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3108 00000100:00080000:0.0:1769076123.112968:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076123.112973:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076123.112980:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076123.112994:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076123.112997:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616505472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076123.113040:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076123.113044:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076123.113048:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076123.113056:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3108 00000100:00080000:0.0:1769076123.113065:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076123.113071:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076123.113078:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076123.113102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076123/real 1769076123] req@ffff995b84642d80 x1855007616505088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076123.113103:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616505600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076123.113145:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076123.113155:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076123.113157:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076123.113163:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3108 00000100:00080000:0.0:1769076123.113168:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076123.113178:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076123.113180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616505088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076123.113183:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076123.113205:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616505728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076123.113215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076123.113221:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076123.113229:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076123.113231:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076123.113239:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076123.113243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616505088:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076123.113251:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3108 00000100:00080000:0.0:1769076123.113262:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076123.113266:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076123.113271:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076123.113283:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616505856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076123.113302:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616505216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076123.113357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076123.113377:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076123/real 1769076123] req@ffff995b84643100 x1855007616505216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076123.113400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616505216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076123.113417:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076123.113424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076123.113431:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616505216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076123.113443:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616505344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076123.113519:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076123.113536:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076123/real 1769076123] req@ffff995b84643480 x1855007616505344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076123.113551:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616505344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076123.113563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076123.113567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076123.113571:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616505344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076123.113579:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616505472:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076123.113600:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076123.113610:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076123/real 1769076123] req@ffff995b84641500 x1855007616505472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076123.113628:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616505472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076123.113644:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076123.113650:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076123.113656:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616505472:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076123.113667:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616505600:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076123.113703:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076123.113724:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076123/real 1769076123] req@ffff995b84641880 x1855007616505600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076123.113753:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616505600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076123.113765:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076123.113768:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076123.113772:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616505600:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076123.113780:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616505728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076123.113804:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076123.113817:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076123/real 1769076123] req@ffff995b84643b80 x1855007616505728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076123.113830:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616505728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076123.113843:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076123.113849:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076123.113855:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616505728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076123.113892:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616505856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076123.113930:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076123.113952:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076123/real 1769076123] req@ffff995b84643800 x1855007616505856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076123.113968:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616505856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076123.113985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076123.113989:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076123.113992:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616505856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076123.816555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076123.816694:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076124.136555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076124.136555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076124.136565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076124.136624:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076124.840538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076124.840744:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076125.160577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076125.160582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076125.160617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076125.160657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076125.864579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076125.864673:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076126.184527:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076126.184574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076126.184576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076126.184608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076126.888512:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076126.888632:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076127.208552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076127.208575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076127.208577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076127.208587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076127.912590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076127.912756:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076128.232642:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076128.232660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076128.232662:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769076128.232663:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076128.232674:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076128.232698:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3113 00000100:00100000:2.0:1769076128.232710:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076128.232719:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076128.232798:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076128.232913:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076128.233032:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616505984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076128.233074:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076128.233080:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076128.233083:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076128.233089:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3113 00000100:00080000:0.0:1769076128.233096:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076128.233103:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076128.233118:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076128.233160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616505984:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076128.233161:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616506112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076128.233190:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076128.233194:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076128.233196:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076128.233201:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3113 00000100:00080000:0.0:1769076128.233207:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076128.233213:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076128.233220:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076128.233253:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616506240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076128.233265:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076128.233269:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076128.233271:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076128.233276:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3113 00000100:00080000:0.0:1769076128.233290:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076128.233295:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076128.233301:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076128.233315:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076128.233323:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616506368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076128.233338:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076128.233343:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076128.233346:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076128.233351:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3113 00000100:00080000:0.0:1769076128.233365:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076128.233370:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076128.233375:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076128.233392:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616506496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076128.233404:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076128.233413:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769076128.233415:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076128/real 1769076128] req@ffff995b84642a00 x1855007616505984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076128.233417:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076128.233450:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3113 00000100:00080000:0.0:1769076128.233459:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076128.233466:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076128.233480:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616505984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076128.233508:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076128.233530:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616506624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076128.233554:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076128.233557:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076128.233564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076128.233568:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076128.233571:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076128.233580:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616505984:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076128.233585:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3113 00000100:00080000:0.0:1769076128.233596:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076128.233601:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076128.233607:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076128.233619:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616506752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076128.233634:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616506112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076128.233677:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076128.233696:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076128/real 1769076128] req@ffff995b84640e00 x1855007616506112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076128.233721:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616506112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076128.233732:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076128.233736:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076128.233740:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616506112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076128.233764:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616506240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076128.233788:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076128.233801:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076128/real 1769076128] req@ffff995b84641180 x1855007616506240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076128.233816:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616506240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076128.233830:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076128.233834:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076128.233837:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616506240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076128.233845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616506368:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076128.233865:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076128.233876:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076128/real 1769076128] req@ffff995b84640000 x1855007616506368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076128.233891:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616506368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076128.233911:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076128.233914:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076128.233951:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616506368:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076128.233977:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616506496:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076128.234001:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076128.234015:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076128/real 1769076128] req@ffff995b84641c00 x1855007616506496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076128.234028:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616506496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076128.234040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076128.234044:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076128.234048:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616506496:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076128.234054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616506624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076128.234073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076128.234083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076128/real 1769076128] req@ffff995b84642680 x1855007616506624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076128.234096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616506624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076128.234106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076128.234109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076128.234113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616506624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076128.234158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616506752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076128.234179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076128.234191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076128/real 1769076128] req@ffff995b84640700 x1855007616506752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076128.234204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616506752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076128.234215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076128.234219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076128.234223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616506752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076128.936544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076128.936797:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076129.256589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076129.256591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076129.256717:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076129.256718:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076129.960611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076129.960742:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076130.280572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076130.280581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076130.280641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076130.280655:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076130.984542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076130.984620:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076131.304625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076131.304631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076131.304671:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076131.304733:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076132.008563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076132.008691:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076132.328540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076132.328579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076132.328657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076132.328658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076133.032607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076133.032798:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076133.352562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076133.352563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076133.352597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076133.352646:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076133.352653:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076133.352658:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076133.352668:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3119 00000100:00100000:2.0:1769076133.352697:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076133.352699:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076133.352712:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076133.352779:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616506880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076133.352811:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076133.352816:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076133.352818:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076133.352823:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3119 00000100:00080000:1.0:1769076133.352828:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076133.352861:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076133.352891:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076133.352915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616506880:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076133.352916:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616507008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076133.352939:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076133.352944:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076133.352946:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076133.352950:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3119 00000100:00080000:1.0:1769076133.352955:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076133.352959:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076133.352964:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076133.352980:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616507136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076133.352991:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076133.352995:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076133.353005:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076133.353011:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3119 00000100:00080000:1.0:1769076133.353017:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076133.353022:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076133.353029:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076133.353048:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616507264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076133.353058:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076133.353061:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076133.353063:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076133.353070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507008:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076133.353091:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3119 00000100:00080000:1.0:1769076133.353102:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076133.353109:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076133.353122:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076133.353142:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616507392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076133.353169:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076133.353175:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769076133.353179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507136:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076133.353180:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076133.353195:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3119 00000100:00080000:1.0:1769076133.353200:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076133.353205:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076133.353213:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076133.353236:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616507520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076133.353248:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076133.353269:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076133.353284:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076133.353288:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3119 00000100:00080000:1.0:1769076133.353292:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076133.353296:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076133.353301:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076133.353313:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616507648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076133.353650:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076133.353710:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076133.353766:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616507520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076133.353801:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616507648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076134.056599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076134.056726:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076134.376586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076134.376636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076134.376679:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076134.376733:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076135.080599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076135.080690:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076135.400602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076135.400646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076135.400654:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076135.400689:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076136.104582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076136.104701:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076136.424624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076136.424631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076136.424642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076136.424680:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:1.0:1769076136.424984:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076136.425039:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076136.425066:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076136.425127:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076136.425143:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076136.425160:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076136.425173:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076136.425190:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076136.425208:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076136.425226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076133/real 1769076136] req@ffff995b868c4a80 x1855007616507392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076136.425273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616507392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076136.425296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076136.425306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076136.425315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076136.425329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076133/real 1769076136] req@ffff995b868c7800 x1855007616507520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076136.425383:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616507520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076136.425400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076136.425407:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076136.425414:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616507520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076136.425427:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076133/real 1769076136] req@ffff995b868c5500 x1855007616507648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076136.425449:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616507648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076136.425521:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076136.425530:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076136.425538:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616507648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076136.425551:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076133/real 1769076136] req@ffff995b868c6300 x1855007616507008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076136.425581:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616507008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076136.425597:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076136.425603:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076136.425609:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076136.425621:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076133/real 1769076136] req@ffff995b868c7100 x1855007616507136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076136.425647:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616507136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076136.425666:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076136.425672:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076136.425933:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076136.425948:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076133/real 1769076136] req@ffff995b868c5c00 x1855007616507264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076136.425975:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616507264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076136.425994:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076136.426000:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076136.426004:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076136.426013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076133/real 1769076136] req@ffff995b868c6d80 x1855007616506880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076136.426029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616506880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076136.426038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076136.426042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076136.426047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616506880:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076137.128590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076137.128696:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076137.448584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076137.448590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076137.448618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076137.448661:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076138.152609:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076138.152733:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076138.472629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076138.472648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076138.472649:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076138.472718:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076138.472725:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076138.472734:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076138.472774:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076138.472795:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3124 00000100:00080000:1.0:1769076138.472813:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076138.472828:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076138.472938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616507776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076138.472983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076138.472988:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076138.472991:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076138.472996:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3124 00000100:00080000:1.0:1769076138.473003:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076138.473008:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076138.473020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076138.473038:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616507904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076138.473052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076138.473057:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076138.473059:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076138.473064:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3124 00000100:00080000:1.0:1769076138.473068:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076138.473072:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076138.473079:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076138.473097:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616508032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076138.473101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616507776:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076138.473126:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076138.473132:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076138.473136:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076138.473146:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3124 00000100:00080000:1.0:1769076138.473153:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076138.473161:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076138.473171:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076138.473195:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616508160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076138.473212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076138.473234:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076138.473242:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076138.473247:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076138.473257:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3124 00000100:00080000:1.0:1769076138.473266:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076138.473273:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076138.473285:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076138.473299:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076138/real 1769076138] req@ffff995b868c6a00 x1855007616507776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076138.473314:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616508288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076138.473352:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616507776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076138.473364:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076138.473371:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076138.473375:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076138.473394:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3124 00000100:00080000:1.0:1769076138.473410:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076138.473423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076138.473426:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076138.473444:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769076138.473445:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076138.473512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616507776:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076138.473524:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616508416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076138.473543:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507904:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076138.473555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076138.473563:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076138.473567:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076138.473578:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3124 00000100:00080000:1.0:1769076138.473587:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076138.473595:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076138.473607:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076138.473634:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616508544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076138.473651:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076138.473886:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076138/real 1769076138] req@ffff995b868c4700 x1855007616507904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076138.473919:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616507904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076138.473937:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076138.473946:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076138.473953:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616507904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076138.473996:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076138.474042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076138.474069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076138/real 1769076138] req@ffff995b868c7480 x1855007616508032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076138.474095:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616508032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076138.474116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076138.474124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076138.474130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076138.474144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076138.474185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076138.474208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076138/real 1769076138] req@ffff995b868c6d80 x1855007616508160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076138.474234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616508160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076138.474252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076138.474258:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076138.474264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076138.474276:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508288:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076138.474317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076138.474339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076138/real 1769076138] req@ffff995b868c5c00 x1855007616508288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076138.474363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616508288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076138.474381:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076138.474387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076138.474393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076138.474405:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616508416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076138.474444:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076138.474513:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076138/real 1769076138] req@ffff995b868c7100 x1855007616508416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076138.474541:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616508416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076138.474572:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076138.474579:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076138.474586:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616508416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076138.474598:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616508544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076138.474640:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076138.474661:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076138/real 1769076138] req@ffff995b868c6300 x1855007616508544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076138.474683:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616508544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076138.474700:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076138.474706:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076138.474713:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616508544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076139.176613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076139.176783:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076139.496580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076139.496582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076139.496650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076139.496698:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076140.200597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076140.200694:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076140.520605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076140.520697:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076140.520740:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076140.520805:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076141.224537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076141.224664:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076141.544550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076141.544557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076141.544577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076141.544609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076142.248552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076142.248623:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076142.568599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076142.568605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076142.568656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076142.568703:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076143.272588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076143.272741:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076143.592614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076143.592614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076143.592637:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076143.592648:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076143.592655:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076143.592696:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3129 00000100:00080000:2.0:1769076143.592713:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076143.592715:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076143.592725:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076143.592726:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076143.592793:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616508672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076143.592845:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076143.592858:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076143.592863:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076143.592874:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3129 00000100:00080000:2.0:1769076143.592883:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076143.592893:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076143.592901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616508672:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769076143.592916:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076143.592948:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616508800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076143.592966:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076143.592970:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076143.592973:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076143.592978:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3129 00000100:00080000:2.0:1769076143.592986:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076143.592990:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076143.592997:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076143.593007:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076143.593017:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616508928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076143.593034:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076143.593039:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076143.593042:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076143.593047:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3129 00000100:00080000:2.0:1769076143.593051:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076143.593056:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076143.593064:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076143.593082:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616509056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076143.593091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076143/real 1769076143] req@ffff995b85ac9880 x1855007616508672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076143.593101:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076143.593117:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076143.593124:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076143.593139:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3129 00000100:00080000:2.0:1769076143.593146:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076143.593151:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076143.593159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616508672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076143.593160:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076143.593184:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616509184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076143.593185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076143.593203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076143.593211:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076143.593218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616508672:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076143.593221:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076143.593229:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076143.593239:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3129 00000100:00080000:2.0:1769076143.593244:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076143.593249:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076143.593255:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076143.593264:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076143.593275:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616509312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076143.593291:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076143.593297:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076143.593300:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076143.593306:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3129 00000100:00080000:2.0:1769076143.593313:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076143.593319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076143.593327:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076143.593336:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076143.593351:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616509440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076143.593370:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076143/real 1769076143] req@ffff995b85acaa00 x1855007616508800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076143.593396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616508800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076143.593408:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076143.593413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076143.593417:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076143.593426:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076143.593449:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076143.593497:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076143/real 1769076143] req@ffff995b85ac9c00 x1855007616508928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076143.593513:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616508928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076143.593535:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076143.593539:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076143.593544:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616508928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076143.593551:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509056:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076143.593574:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076143.593586:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076143/real 1769076143] req@ffff995b85acb800 x1855007616509056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076143.593601:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616509056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076143.593612:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076143.593616:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076143.593619:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076143.593627:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509184:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076143.593644:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076143.593656:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076143/real 1769076143] req@ffff995b85aca300 x1855007616509184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076143.593707:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616509184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076143.593719:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076143.593723:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076143.593728:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076143.593752:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616509312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076143.593781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076143.593795:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076143/real 1769076143] req@ffff995b85ac8a80 x1855007616509312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076143.593810:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616509312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076143.593820:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076143.593824:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076143.593828:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616509312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076143.593835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616509440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076143.593853:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076143.593864:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076143/real 1769076143] req@ffff995b85acb100 x1855007616509440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076143.593878:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616509440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076143.593888:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076143.593891:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076143.593895:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616509440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076144.296572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076144.296724:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076144.616604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076144.616636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076144.616684:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076144.616705:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076145.320544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076145.320845:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076145.640669:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076145.640864:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076145.640951:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076145.640962:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076146.344689:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076146.344813:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076146.664591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076146.664673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076146.664890:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076146.664923:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076147.368575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076147.368766:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076147.688523:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076147.688615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076147.688640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076147.688641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076148.392558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076148.392736:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076148.712589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076148.712625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076148.712652:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076148.712660:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769076148.712663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076148.712665:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076148.712667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076148.712678:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3134 00000100:00080000:2.0:1769076148.712691:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076148.712699:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076148.712735:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616509568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076148.712759:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076148.712763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076148.712764:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076148.712769:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3134 00000100:00080000:2.0:1769076148.712788:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076148.712793:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076148.712801:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076148.712813:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616509696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076148.712822:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076148.712825:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076148.712827:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076148.712830:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3134 00000100:00080000:2.0:1769076148.712832:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076148.712835:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076148.712838:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076148.712849:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616509824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076148.712858:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076148.712862:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076148.712865:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076148.712868:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616509568:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076148.712872:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3134 00000100:00080000:2.0:1769076148.712878:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076148.712884:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076148.712888:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076148.712899:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616509952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076148.712908:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076148.712912:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076148.712914:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076148.712917:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3134 00000100:00080000:2.0:1769076148.712939:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076148.712946:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076148.712953:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076148.712955:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076148.712980:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616510080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076148.712995:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076148.712999:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076148.713002:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076148.713007:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3134 00000100:00080000:2.0:1769076148.713014:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076148.713022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076148/real 1769076148] req@ffff995b85ac8380 x1855007616509568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076148.713029:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076148.713038:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076148.713060:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616510208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076148.713064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616509568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076148.713098:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076148.713100:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076148.713108:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076148.713113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076148.713115:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076148.713124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616509568:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076148.713128:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3134 00000100:00080000:2.0:1769076148.713135:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076148.713142:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076148.713152:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076148.713158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076148.713184:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616510336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076148.713211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076148.713232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076148/real 1769076148] req@ffff995b85acbb80 x1855007616509696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076148.713249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616509696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076148.713260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076148.713264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076148.713269:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076148.713298:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076148.713330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076148.713343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076148/real 1769076148] req@ffff995b85acb100 x1855007616509824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076148.713358:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616509824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076148.713372:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076148.713376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076148.713380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076148.713388:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509952:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076148.713409:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076148.713439:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076148/real 1769076148] req@ffff995b85ac8a80 x1855007616509952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076148.713464:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616509952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076148.713476:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076148.713481:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076148.713514:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616509952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076148.713523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510080:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076148.713548:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076148.713563:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076148/real 1769076148] req@ffff995b85aca300 x1855007616510080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076148.713624:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616510080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076148.713636:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076148.713640:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076148.713645:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076148.713652:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616510208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076148.713676:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076148.713689:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076148/real 1769076148] req@ffff995b85acb800 x1855007616510208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076148.713704:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616510208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076148.713713:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076148.713717:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076148.713721:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616510208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076148.713728:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616510336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076148.713746:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076148.713833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076148/real 1769076148] req@ffff995b85ac9c00 x1855007616510336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076148.713849:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616510336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076148.713866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076148.713870:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076148.713874:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616510336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076149.416644:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076149.416724:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076149.736603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076149.736640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076149.736662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076149.736717:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076150.440540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076150.440704:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076150.760638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076150.760649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076150.760735:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076150.760777:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076151.464653:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076151.464729:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076151.784654:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076151.784684:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076151.784703:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076151.784732:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076152.488578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076152.488758:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076152.808656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076152.808676:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076152.808711:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076152.808762:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076153.512570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076153.512732:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076153.832653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076153.832682:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076153.832723:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076153.832724:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076153.832732:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076153.832753:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076153.832763:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076153.832785:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3139 00000100:00080000:0.0:1769076153.832809:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076153.832824:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076153.832900:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616510464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076153.832949:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076153.832955:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076153.832958:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076153.832964:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3139 00000100:00080000:0.0:1769076153.832970:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076153.832977:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076153.832993:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076153.833032:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616510592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076153.833033:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616510464:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076153.833072:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076153.833079:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076153.833083:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076153.833095:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3139 00000100:00080000:0.0:1769076153.833102:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076153.833108:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076153.833115:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076153.833130:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616510720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076153.833143:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076153.833148:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076153.833152:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769076153.833167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076153.833175:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3139 00000100:00080000:0.0:1769076153.833183:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076153.833192:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076153.833200:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076153.833222:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616510848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076153.833233:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076153.833238:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076153.833241:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076153.833248:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3139 00000100:00080000:0.0:1769076153.833253:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076153.833266:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076153.833266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076153/real 1769076153] req@ffff995b84640380 x1855007616510464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076153.833276:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076153.833303:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616510976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076153.833332:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076153.833334:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616510464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076153.833353:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076153.833366:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076153.833388:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3139 00000100:00080000:3.0:1769076153.833391:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076153.833395:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076153.833402:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076153.833410:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076153.833428:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076153.833428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616510464:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076153.833539:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616511104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076153.833568:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076153.833577:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076153.833587:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076153.833587:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510592:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076153.833619:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3139 00000100:00080000:0.0:1769076153.833629:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076153.833638:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076153.833649:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076153.833659:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076153.833674:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616511232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076153.833692:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076153/real 1769076153] req@ffff995b84641f80 x1855007616510592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076153.833727:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616510592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076153.833741:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076153.833746:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076153.833754:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076153.833765:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076153.833788:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076153.833800:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076153/real 1769076153] req@ffff995b84640a80 x1855007616510720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076153.833815:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616510720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076153.833827:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076153.833831:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076153.833835:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076153.833843:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076153.833862:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076153.833874:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076153/real 1769076153] req@ffff995b84640700 x1855007616510848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076153.833888:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616510848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076153.833899:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076153.833902:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076153.833907:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076153.833914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076153.833932:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076153.833943:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076153/real 1769076153] req@ffff995b84642680 x1855007616510976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076153.833957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616510976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076153.833967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076153.833971:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076153.833975:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616510976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076153.833982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616511104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076153.834000:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076153.834012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076153/real 1769076153] req@ffff995b84641c00 x1855007616511104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076153.834026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616511104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076153.834036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076153.834040:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076153.834043:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616511104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076153.834072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616511232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076153.834091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076153.834104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076153/real 1769076153] req@ffff995b84640000 x1855007616511232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076153.834118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616511232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076153.834128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076153.834132:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076153.834135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616511232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076154.536596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076154.536701:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076154.856596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076154.856634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076154.856662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076154.856703:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076155.560523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076155.560600:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076155.880631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076155.880636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076155.880638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076155.880638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076156.584630:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076156.584722:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076156.904614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076156.904628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076156.904654:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076156.904679:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076157.608597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076157.608780:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076157.928643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076157.928650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076157.928651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076157.928654:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076158.632539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076158.632701:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076158.952548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076158.952583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076158.952615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076158.952631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076158.952641:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076158.952650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076158.952656:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076158.952671:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3144 00000100:00080000:3.0:1769076158.952687:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076158.952698:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076158.952747:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616511360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076158.952785:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076158.952789:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076158.952791:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076158.952797:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3144 00000100:00080000:3.0:1769076158.952802:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076158.952808:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076158.952819:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076158.952829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616511360:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076158.952840:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616511488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076158.952859:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076158.952864:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076158.952868:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076158.952874:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3144 00000100:00080000:3.0:1769076158.952894:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076158.952901:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076158.952908:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076158.952915:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076158.952938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616511616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076158.952948:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076158.952952:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076158.952954:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076158.952959:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3144 00000100:00080000:3.0:1769076158.952964:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076158.952967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076158/real 1769076158] req@ffff995b84558000 x1855007616511360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076158.952973:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076158.952985:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076158.953012:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616511744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076158.953015:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616511360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076158.953077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076158.953081:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076158.953090:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076158.953092:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076158.953101:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076158.953104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616511360:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076158.953111:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3144 00000100:00080000:3.0:1769076158.953122:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076158.953129:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076158.953137:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076158.953159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616511488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076158.953160:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616511872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076158.953184:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076158.953188:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076158.953191:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076158.953197:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3144 00000100:00080000:3.0:1769076158.953207:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076158.953214:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076158.953221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076158.953225:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076158.953258:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616512000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076158.953262:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076158/real 1769076158] req@ffff995b8455a680 x1855007616511488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076158.953295:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076158.953302:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076158.953303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616511488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076158.953305:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076158.953317:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3144 00000100:00080000:3.0:1769076158.953327:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076158.953328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076158.953335:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076158.953339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769076158.953343:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076158.953346:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616511488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076158.953358:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616511616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076158.953362:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616512128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076158.953400:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076158.953449:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076158/real 1769076158] req@ffff995b84558a80 x1855007616511616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076158.953489:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616511616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076158.953507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076158.953513:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076158.953518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616511616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076158.953527:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616511744:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076158.953559:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076158.953576:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076158/real 1769076158] req@ffff995b84559500 x1855007616511744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076158.953595:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616511744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076158.953609:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076158.953612:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076158.953615:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616511744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076158.953634:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616511872:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076158.953655:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076158.953666:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076158/real 1769076158] req@ffff995b8455aa00 x1855007616511872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076158.953677:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616511872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076158.953686:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076158.953689:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076158.953692:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616511872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076158.953698:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616512000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076158.953712:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076158.953720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076158/real 1769076158] req@ffff995b8455bb80 x1855007616512000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076158.953730:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616512000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076158.953738:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076158.953740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076158.953743:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616512000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076158.953748:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616512128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076158.953760:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076158.953769:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076158/real 1769076158] req@ffff995b84558000 x1855007616512128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076158.953779:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616512128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076158.953786:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076158.953789:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076158.953792:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616512128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076159.656519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076159.656579:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076159.976558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076159.976583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076159.976589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076159.976602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076160.680562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076160.680697:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076161.000587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076161.000596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076161.000636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076161.000660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076161.704574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076161.704668:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076162.024564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076162.024604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076162.024612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076162.024648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076162.728561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076162.728723:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076163.048564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076163.048618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076163.048619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076163.048663:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076163.752582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076163.752700:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076164.072583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076164.072605:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076164.072610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076164.072619:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769076164.072622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076164.072627:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076164.072645:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3149 00000100:00100000:2.0:1769076164.072656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076164.072663:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076164.072678:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076164.072735:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616512256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076164.072775:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076164.072780:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076164.072782:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076164.072788:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3149 00000100:00080000:0.0:1769076164.072793:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076164.072799:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076164.072810:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076164.072831:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616512384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076164.072842:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616512256:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076164.072867:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076164.072874:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076164.072879:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076164.072889:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3149 00000100:00080000:0.0:1769076164.072897:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076164.072906:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076164.072917:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076164.072937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076164.072945:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616512512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076164.072961:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076164.072966:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076164.072968:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076164.072974:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3149 00000100:00080000:0.0:1769076164.072980:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076164.072985:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076164.072992:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076164.072994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076164/real 1769076164] req@ffff995b84641180 x1855007616512256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076164.073013:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616512640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076164.073027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616512256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076164.073030:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076164.073046:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076164.073051:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076164.073065:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3149 00000100:00080000:2.0:1769076164.073067:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076164.073071:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076164.073079:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076164.073081:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076164.073091:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076164.073093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616512256:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076164.073109:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616512768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076164.073122:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076164.073126:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076164.073132:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076164.073133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616512384:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076164.073145:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3149 00000100:00080000:0.0:1769076164.073152:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076164.073157:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076164.073165:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076164.073181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076164.073188:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616512896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076164.073205:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076164.073206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076164/real 1769076164] req@ffff995b84640e00 x1855007616512384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076164.073214:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076164.073220:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076164.073235:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3149 00000100:00080000:0.0:1769076164.073246:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076164.073255:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076164.073256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616512384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076164.073261:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076164.073275:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616513024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076164.073364:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076164.073372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076164.073377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616512384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076164.073389:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616512512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076164.073418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076164.073432:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076164/real 1769076164] req@ffff995b84642a00 x1855007616512512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076164.073446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616512512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076164.073490:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076164.073494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076164.073498:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616512512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076164.073506:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616512640:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076164.073529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076164.073542:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076164/real 1769076164] req@ffff995b84643800 x1855007616512640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076164.073556:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616512640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076164.073566:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076164.073570:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076164.073574:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616512640:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076164.073581:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616512768:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076164.073598:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076164.073608:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076164/real 1769076164] req@ffff995b84643b80 x1855007616512768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076164.073620:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616512768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076164.073631:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076164.073634:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076164.073638:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616512768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076164.073659:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616512896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076164.073684:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076164.073704:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076164/real 1769076164] req@ffff995b84641880 x1855007616512896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076164.073727:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616512896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076164.073743:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076164.073750:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076164.073757:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616512896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076164.073768:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616513024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076164.073803:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076164.073819:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076164/real 1769076164] req@ffff995b84641500 x1855007616513024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076164.073833:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616513024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076164.073843:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076164.073847:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076164.073851:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616513024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076164.776586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076164.776693:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076165.096622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076165.096626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076165.096692:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076165.096730:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076165.800593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076165.800708:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076166.120542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076166.120549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076166.120604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076166.120626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076166.824620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076166.824889:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076167.144625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076167.144640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076167.144640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076167.144712:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076167.848531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076167.848624:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076168.168584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076168.168610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076168.168700:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076168.168704:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076168.872573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076168.872690:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076169.192572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076169.192605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076169.192648:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076169.192660:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769076169.192660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076169.192670:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076169.192687:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3154 00000100:00100000:2.0:1769076169.192687:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076169.192709:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076169.192723:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076169.192774:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616513152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076169.192814:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076169.192819:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076169.192822:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076169.192829:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3154 00000100:00080000:0.0:1769076169.192836:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076169.192843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076169.192860:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076169.192875:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616513152:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076169.192899:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616513280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076169.192929:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076169.192937:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076169.192942:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076169.192953:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3154 00000100:00080000:0.0:1769076169.192967:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076169.192975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076169.192985:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076169.193000:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076169.193033:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616513408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076169.193054:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076169.193061:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1769076169.193068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076169/real 1769076169] req@ffff995b84643480 x1855007616513152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076169.193070:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076169.193092:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3154 00000100:00080000:0.0:1769076169.193106:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076169.193114:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616513152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076169.193120:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076169.193139:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769076169.193163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076169.193174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076169.193182:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616513536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076169.193190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616513152:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076169.193222:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076169.193259:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076169.193265:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076169.193269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616513280:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076169.193281:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3154 00000100:00080000:0.0:1769076169.193293:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076169.193304:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076169.193314:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769076169.193318:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076169.193338:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076169/real 1769076169] req@ffff995b84643100 x1855007616513280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076169.193355:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616513664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076169.193366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616513280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076169.193387:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076169.193391:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076169.193396:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076169.193403:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076169.193406:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076169.193415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616513280:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076169.193426:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3154 00000100:00100000:2.0:1769076169.193434:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616513408:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076169.193441:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076169.193506:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076169.193507:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769076169.193528:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076169.193543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076169/real 1769076169] req@ffff995b84642d80 x1855007616513408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076169.193571:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616513792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076169.193593:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616513408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076169.193600:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076169.193618:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076169.193622:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076169.193625:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076169.193636:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076169.193644:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3154 00000100:00100000:2.0:1769076169.193653:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616513408:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076169.193659:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076169.193671:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616513536:192.168.204.132@tcp:8:kworker.0 00000100:00080000:0.0:1769076169.193673:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076169.193690:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076169.193712:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076169.193720:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616513920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076169.193737:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076169/real 1769076169] req@ffff995b84641500 x1855007616513536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076169.193756:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616513536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076169.193768:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076169.193774:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076169.193779:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616513536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076169.193803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616513664:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076169.193823:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076169.193835:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076169/real 1769076169] req@ffff995b84641880 x1855007616513664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076169.193849:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616513664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076169.193860:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076169.193863:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076169.193867:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616513664:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076169.193873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616513792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076169.193892:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076169.193903:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076169/real 1769076169] req@ffff995b84643b80 x1855007616513792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076169.193918:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616513792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076169.193928:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076169.193931:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076169.193935:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616513792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076169.193941:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616513920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076169.193958:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076169.193968:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076169/real 1769076169] req@ffff995b84643800 x1855007616513920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076169.193981:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616513920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076169.193991:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076169.193995:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076169.193999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616513920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076169.896605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076169.896724:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076170.216613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076170.216619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076170.216624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076170.216626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076170.920547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076170.920714:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076171.240538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076171.240539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076171.240546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076171.240552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076171.944580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076171.944671:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076172.264589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076172.264615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076172.264621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076172.264622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076172.968562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076172.968689:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076173.288572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076173.288575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076173.288582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076173.288621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076173.992532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076173.992595:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076174.312523:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076174.312531:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076174.312536:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076174.312538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076174.312541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076174.312544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076174.312545:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3160 00000100:00080000:3.0:1769076174.312558:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076174.312564:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076174.312580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076174.312596:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616514048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076174.312618:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076174.312621:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076174.312622:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076174.312625:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3160 00000100:00080000:3.0:1769076174.312628:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076174.312631:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076174.312639:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076174.312649:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616514048:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076174.312650:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616514176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076174.312661:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076174.312664:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076174.312665:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076174.312667:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3160 00000100:00080000:3.0:1769076174.312670:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076174.312673:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076174.312676:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076174.312684:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616514304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076174.312691:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076174.312693:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076174.312694:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076174.312697:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3160 00000100:00080000:3.0:1769076174.312700:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076174.312703:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076174.312707:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076174.312713:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076174.312716:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616514432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076174.312723:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076174.312726:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076174.312727:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076174.312730:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3160 00000100:00080000:3.0:1769076174.312733:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076174.312735:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076174.312737:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076174.312744:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616514560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076174.312750:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076174.312752:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076174/real 1769076174] req@ffff995b84558700 x1855007616514048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076174.312753:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076174.312757:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076174.312765:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3160 00000100:00080000:3.0:1769076174.312769:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076174.312771:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076174.312775:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076174.312776:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616514048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076174.312783:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616514688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076174.312786:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076174.312795:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076174.312803:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616514048:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076174.312805:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076174.312809:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076174.312810:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076174.312813:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3160 00000100:00080000:3.0:1769076174.312816:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076174.312818:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076174.312820:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076174.312827:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616514176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076174.312832:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616514816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076174.312849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076174.312859:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076174/real 1769076174] req@ffff995b84559c00 x1855007616514176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076174.312868:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616514176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076174.312874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076174.312877:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076174.312879:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616514176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076174.312883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616514304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076174.312893:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076174.312900:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076174/real 1769076174] req@ffff995b8455b100 x1855007616514304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076174.312907:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616514304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076174.312914:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076174.312916:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076174.312917:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616514304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076174.312921:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616514432:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076174.312931:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076174.312937:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076174/real 1769076174] req@ffff995b8455a300 x1855007616514432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076174.312944:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616514432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076174.312950:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076174.312952:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076174.312954:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616514432:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076174.312957:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616514560:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076174.312966:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076174.312972:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076174/real 1769076174] req@ffff995b8455ad80 x1855007616514560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076174.312979:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616514560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076174.312986:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076174.312988:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076174.312990:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616514560:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076174.312993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616514688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076174.313002:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076174.313008:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076174/real 1769076174] req@ffff995b84559880 x1855007616514688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076174.313015:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616514688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076174.313020:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076174.313022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076174.313024:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616514688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076174.313035:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616514816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076174.313044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076174.313050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076174/real 1769076174] req@ffff995b8455b480 x1855007616514816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076174.313056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616514816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076174.313062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076174.313064:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076174.313066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616514816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076175.016538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076175.016713:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076175.336570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076175.336639:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076175.336650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076175.336656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076176.040640:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076176.040715:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076176.360615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076176.360620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076176.360643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076176.360667:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076177.064582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076177.064717:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076177.384597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076177.384619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076177.384697:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076177.384730:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076178.088593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076178.088768:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076178.408586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076178.408602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076178.408618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076178.408630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076179.112561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076179.112732:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076179.432620:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076179.432629:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076179.432641:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076179.432653:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076179.432673:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3165 00000100:00100000:0.0:1769076179.432678:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076179.432684:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076179.432696:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076179.432716:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076179.432758:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076179.432788:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616514944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076179.432846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076179.432852:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076179.432855:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076179.432864:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3165 00000100:00080000:2.0:1769076179.432870:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076179.432877:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076179.432899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616514944:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769076179.432900:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076179.432927:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616515072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076179.432943:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076179.432947:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076179.432950:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076179.432956:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3165 00000100:00080000:2.0:1769076179.432962:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076179.432966:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076179.432972:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076179.432989:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616515200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076179.433002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076179.433007:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076179.433010:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076179.433014:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3165 00000800:00000100:3.0:1769076179.433022:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076179.433023:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076179.433035:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076179.433042:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076179.433086:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616515328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076179.433101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076179.433105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076179.433108:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076179.433115:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3165 00000100:00080000:2.0:1769076179.433122:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076179.433129:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076179.433139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076179/real 1769076179] req@ffff995b85acaa00 x1855007616514944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076179.433140:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076179.433169:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616515456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076179.433181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616514944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076179.433195:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076179.433212:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076179.433219:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076179.433223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076179.433229:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3165 00000100:00080000:2.0:1769076179.433236:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076179.433238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076179.433246:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076179.433253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616514944:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769076179.433257:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076179.433279:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616515584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076179.433292:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076179.433297:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076179.433304:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076179.433305:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515072:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076179.433314:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3165 00000100:00080000:2.0:1769076179.433319:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076179.433325:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076179.433336:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076179.433361:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616515712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076179.433375:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076179.433402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076179/real 1769076179] req@ffff995b85ac9880 x1855007616515072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076179.433423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616515072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076179.433434:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076179.433440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076179.433445:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076179.433486:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076179.433512:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076179.433527:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076179/real 1769076179] req@ffff995b85ac9c00 x1855007616515200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076179.433542:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616515200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076179.433555:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076179.433559:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076179.433562:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076179.433570:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515328:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076179.433592:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076179.433604:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076179/real 1769076179] req@ffff995b85acb800 x1855007616515328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076179.433619:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616515328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076179.433631:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076179.433634:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076179.433638:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515328:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076179.433645:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515456:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076179.433664:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076179.433675:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076179/real 1769076179] req@ffff995b85aca300 x1855007616515456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076179.433689:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616515456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076179.433700:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076179.433704:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076179.433707:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076179.433713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616515584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076179.433731:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076179.433742:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076179/real 1769076179] req@ffff995b85ac8a80 x1855007616515584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076179.433755:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616515584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076179.433773:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076179.433777:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076179.433781:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616515584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076179.433803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616515712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076179.433821:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076179.433833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076179/real 1769076179] req@ffff995b85acb100 x1855007616515712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076179.433847:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616515712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076179.433857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076179.433861:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076179.433865:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616515712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076180.136576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076180.136799:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076180.456596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076180.456653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076180.456668:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076180.456704:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076181.160594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076181.160730:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076181.480601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076181.480630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076181.480687:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076181.480693:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076182.184557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076182.184695:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076182.504538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076182.504541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076182.504546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076182.504586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076183.208569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076183.208683:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076183.528573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076183.528624:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076183.528647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076183.528710:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076184.232564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076184.232738:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076184.552546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076184.552550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076184.552552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076184.552574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076184.552587:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076184.552595:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076184.552599:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076184.552609:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3170 00000100:00080000:1.0:1769076184.552621:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076184.552629:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076184.552666:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616515840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076184.552695:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076184.552699:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076184.552701:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076184.552706:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3170 00000100:00080000:1.0:1769076184.552711:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076184.552716:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076184.552727:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076184.552729:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616515840:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076184.552745:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616515968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076184.552757:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076184.552761:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076184.552764:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076184.552768:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3170 00000100:00080000:1.0:1769076184.552773:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076184.552778:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076184.552783:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076184.552801:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076184.552809:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616516096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076184.552822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076184.552826:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076184.552829:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076184.552834:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3170 00000100:00080000:1.0:1769076184.552840:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076184.552843:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076184/real 1769076184] req@ffff995b868c5500 x1855007616515840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076184.552849:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076184.552860:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076184.552880:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616516224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076184.552881:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616515840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076184.552914:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076184.552916:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076184.552920:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076184.552924:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769076184.552927:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616515840:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076184.552929:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076184.552936:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3170 00000100:00080000:1.0:1769076184.552941:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076184.552946:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076184.552950:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515968:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769076184.552952:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076184.552985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076184.552995:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616516352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076184.553002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076184/real 1769076184] req@ffff995b868c7800 x1855007616515968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076184.553023:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076184.553027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616515968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076184.553031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076184.553036:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076184.553043:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076184.553047:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3170 00000100:00080000:3.0:1769076184.553051:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076184.553056:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076184.553056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616515968:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076184.553067:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076184.553068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516096:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769076184.553079:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076184.553091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076184.553098:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616516480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076184.553102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076184/real 1769076184] req@ffff995b868c4a80 x1855007616516096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076184.553128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616516096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076184.553130:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076184.553140:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076184.553144:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076184.553154:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3170 00000100:00080000:3.0:1769076184.553155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076184.553162:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076184.553162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076184.553170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516096:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076184.553174:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076184.553179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516224:192.168.204.132@tcp:8:kworker.0 00000008:00100000:1.0:1769076184.553184:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076184.553196:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076184.553203:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616516608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076184.553207:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076184/real 1769076184] req@ffff995b868c6300 x1855007616516224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076184.553221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616516224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076184.553244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076184.553248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076184.553253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516224:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076184.553274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516352:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076184.553292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076184.553301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076184/real 1769076184] req@ffff995b868c7100 x1855007616516352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076184.553309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616516352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076184.553315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076184.553317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076184.553320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076184.553324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616516480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076184.553334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076184.553339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076184/real 1769076184] req@ffff995b868c5c00 x1855007616516480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076184.553347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616516480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076184.553352:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076184.553354:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076184.553356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616516480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076184.553359:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616516608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076184.553368:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076184.553374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076184/real 1769076184] req@ffff995b868c6d80 x1855007616516608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076184.553381:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616516608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076184.553386:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076184.553388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076184.553390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616516608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076185.256581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076185.256664:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076185.576559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076185.576560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076185.576577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076185.576630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076186.280557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076186.280657:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076186.600588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076186.600607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076186.600680:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076186.600684:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076187.304567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076187.304782:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076187.624542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076187.624556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076187.624575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076187.624575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076188.328511:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076188.328572:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076188.648538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076188.648544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076188.648551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076188.648565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076189.352578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076189.352751:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076189.672582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076189.672588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076189.672617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076189.672663:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076189.672673:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076189.672679:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076189.672692:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3175 00000100:00100000:1.0:1769076189.672701:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076189.672706:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076189.672720:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076189.672766:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616516736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076189.672803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076189.672809:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076189.672811:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076189.672817:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3175 00000100:00080000:0.0:1769076189.672821:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076189.672827:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076189.672838:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076189.672854:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616516864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076189.672870:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076189.672877:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616516736:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076189.672878:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076189.672884:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076189.672918:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3175 00000100:00080000:0.0:1769076189.672923:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076189.672929:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076189.672936:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076189.672955:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616516992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076189.672969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076189.672973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076189.672975:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076189.672981:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3175 00000100:00080000:0.0:1769076189.672985:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076189.672989:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076189.672996:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076189.673011:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616517120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076189.673020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076189.673037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076189.673042:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076189.673045:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076189.673052:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3175 00000100:00080000:0.0:1769076189.673057:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076189.673061:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076189.673067:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076189.673089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076189/real 1769076189] req@ffff995b84642a00 x1855007616516736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076189.673090:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616517248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076189.673152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076189.673153:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616516736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076189.673168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076189.673177:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076189.673192:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3175 00000100:00080000:3.0:1769076189.673196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076189.673198:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076189.673211:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076189.673217:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076189.673219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076189.673234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616516736:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076189.673236:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616517376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076189.673252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076189.673256:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076189.673258:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076189.673270:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3175 00000100:00080000:0.0:1769076189.673275:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076189.673279:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076189.673284:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076189.673295:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616517504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076189.673313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076189.673361:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076189.673381:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076189/real 1769076189] req@ffff995b84640e00 x1855007616516864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076189.673398:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616516864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076189.673410:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076189.673415:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076189.673420:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076189.673428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076189.673449:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076189.673501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076189/real 1769076189] req@ffff995b84641180 x1855007616516992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076189.673517:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616516992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076189.673530:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076189.673534:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076189.673538:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616516992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076189.673547:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616517120:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076189.673571:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076189.673583:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076189/real 1769076189] req@ffff995b84640000 x1855007616517120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076189.673597:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616517120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076189.673608:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076189.673612:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076189.673615:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616517120:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076189.673622:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616517248:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076189.673640:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076189.673651:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076189/real 1769076189] req@ffff995b84641c00 x1855007616517248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076189.673686:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616517248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076189.673704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076189.673710:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076189.673716:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616517248:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076189.673729:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616517376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076189.673770:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076189.673793:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076189/real 1769076189] req@ffff995b84642680 x1855007616517376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076189.673810:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616517376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076189.673821:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076189.673825:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076189.673829:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616517376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076189.673857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616517504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076189.673883:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076189.673936:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076189/real 1769076189] req@ffff995b84640700 x1855007616517504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076189.673952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616517504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076189.673972:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076189.673976:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076189.673980:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616517504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076190.376523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076190.376592:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076190.696622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076190.696658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076190.696664:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076190.696723:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076191.400582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076191.400717:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076191.720596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076191.720614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076191.720630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076191.720642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076192.424596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076192.424663:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076192.744586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076192.744636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076192.744641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076192.744655:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076193.448583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076193.448663:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076193.768590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076193.768593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076193.768598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076193.768612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076194.472525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076194.472628:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076194.792520:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076194.792528:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076194.792530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076194.792535:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076194.792542:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076194.792546:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076194.792553:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3180 00000100:00100000:2.0:1769076194.792563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076194.792570:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076194.792578:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076194.792611:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616517632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076194.792633:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076194.792635:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076194.792637:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076194.792640:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3180 00000100:00080000:0.0:1769076194.792642:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076194.792645:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076194.792653:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076194.792663:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616517760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076194.792666:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616517632:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076194.792680:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076194.792683:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076194.792684:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076194.792687:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3180 00000100:00080000:0.0:1769076194.792689:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076194.792692:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076194.792695:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076194.792703:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616517888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076194.792710:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076194.792713:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076194.792714:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076194.792716:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3180 00000100:00080000:0.0:1769076194.792718:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076194.792720:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076194.792723:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076194.792731:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616518016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076194.792737:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076194.792741:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076194.792744:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076194.792745:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076194.792749:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3180 00000100:00080000:0.0:1769076194.792752:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076194.792755:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076194.792758:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076194.792765:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616518144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076194.792771:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076194.792774:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076194.792777:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076194.792778:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076194/real 1769076194] req@ffff995b84640a80 x1855007616517632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076194.792784:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3180 00000100:00080000:0.0:1769076194.792790:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076194.792796:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076194.792818:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616517632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076194.792820:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076194.792832:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769076194.792834:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616518272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076194.792839:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076194.792846:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616517632:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076194.792849:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076194.792852:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076194.792854:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076194.792858:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3180 00000100:00080000:0.0:1769076194.792860:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076194.792863:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076194.792866:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076194.792871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616517760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076194.792875:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616518400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076194.792896:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076194.792909:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076194/real 1769076194] req@ffff995b84641f80 x1855007616517760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076194.792922:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616517760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076194.792928:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076194.792930:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076194.792932:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616517760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076194.792936:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616517888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076194.792947:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076194.792953:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076194/real 1769076194] req@ffff995b84640380 x1855007616517888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076194.792960:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616517888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076194.792967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076194.792969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076194.792971:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616517888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076194.792974:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518016:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076194.792984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076194.792990:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076194/real 1769076194] req@ffff995b84640700 x1855007616518016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076194.792997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616518016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076194.793008:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076194.793010:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076194.793012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518016:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076194.793015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518144:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076194.793024:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076194.793029:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076194/real 1769076194] req@ffff995b84642680 x1855007616518144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076194.793036:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616518144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076194.793042:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076194.793044:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076194.793046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518144:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076194.793049:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616518272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076194.793058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076194.793064:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076194/real 1769076194] req@ffff995b84641c00 x1855007616518272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076194.793071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616518272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076194.793076:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076194.793078:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076194.793081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616518272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076194.793094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616518400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076194.793112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076194.793123:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076194/real 1769076194] req@ffff995b84640000 x1855007616518400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076194.793137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616518400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076194.793146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076194.793150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076194.793154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616518400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076195.496572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076195.496708:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076195.816583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076195.816667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076195.816672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076195.816686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076196.520567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076196.520699:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076196.840590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076196.840591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076196.840636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076196.840942:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076197.544600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076197.544743:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076197.864658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076197.864720:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076197.864827:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076197.864834:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076198.568599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076198.568728:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076198.888615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076198.888645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076198.888671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076198.888697:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076199.592578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076199.592714:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076199.912645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076199.912683:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076199.912748:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076199.912771:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076199.912782:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769076199.912783:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076199.912790:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076199.912809:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3185 00000100:00080000:1.0:1769076199.912830:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076199.912843:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076199.912917:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616518528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076199.912958:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076199.912963:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076199.912966:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076199.912972:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3185 00000100:00080000:1.0:1769076199.912977:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076199.912984:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076199.913009:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076199.913023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616518528:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076199.913032:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616518656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076199.913050:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076199.913056:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076199.913061:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076199.913069:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3185 00000100:00080000:1.0:1769076199.913087:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076199.913095:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076199.913105:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076199.913132:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616518784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076199.913153:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076199.913161:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076199.913166:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076199.913174:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3185 00000100:00080000:1.0:1769076199.913184:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076199.913193:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076199.913204:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076199.913241:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616518912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076199.913269:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076199.913270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518656:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076199.913288:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076199.913293:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076199.913301:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3185 00000100:00080000:1.0:1769076199.913307:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076199.913312:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076199.913322:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076199.913357:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616519040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076199.913357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518784:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076199.913389:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076199.913394:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076199.913397:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076199.913405:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3185 00000100:00080000:1.0:1769076199.913410:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076199.913416:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076199.913423:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076199.913446:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616519168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076199.913509:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518912:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769076199.913531:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076199.913538:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076199.913542:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076199.913549:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3185 00000100:00080000:1.0:1769076199.913562:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076199.913568:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076199.913577:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076199.913584:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076199.913603:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616519296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076199.913676:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616519168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076199.913768:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616519296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076200.616617:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076200.616831:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076200.936670:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076200.936678:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076200.936678:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076200.936825:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076201.640509:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076201.640676:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076201.960591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076201.960593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076201.960647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076201.960648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076202.664612:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076202.664762:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076202.984622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076202.984738:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076202.984814:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076202.984933:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000400:00000100:1.0:1769076202.985099:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076202.985192:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076202.985222:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076202.985290:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076202.985320:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076202.985373:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076199/real 1769076202] req@ffff995b868c4700 x1855007616518656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1769076202.985387:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076202.985434:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076202.985497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616518656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076202.985530:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076202.985547:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000800:00000100:1.0:1769076202.985559:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076202.985572:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076202.985593:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076202.985604:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076199/real 1769076202] req@ffff995b868c6a00 x1855007616518784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076202.985647:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616518784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076202.985693:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076202.985701:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076202.985709:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076202.985725:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076199/real 1769076202] req@ffff995b868c7480 x1855007616518528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076202.985765:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616518528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076202.985780:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076202.985787:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076202.985793:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616518528:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076202.986373:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076199/real 1769076202] req@ffff995b868c7100 x1855007616519168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076202.986407:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616519168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076202.986426:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076202.986434:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076202.986441:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616519168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076202.986520:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076199/real 1769076202] req@ffff995b868c6300 x1855007616519296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076202.986557:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616519296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076202.986576:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076202.986602:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076202.986609:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616519296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076202.986626:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076199/real 1769076202] req@ffff995b868c6d80 x1855007616518912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076202.986651:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616518912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076202.986691:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076202.986698:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076202.986705:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616518912:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076202.986717:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076199/real 1769076202] req@ffff995b868c5c00 x1855007616519040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076202.986748:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616519040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076202.986763:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076202.986768:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076202.986774:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076203.688579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076203.688734:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076204.008536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076204.008561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076204.008587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076204.008613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076204.712595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076204.712732:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076205.032663:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076205.032667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076205.032682:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076205.032689:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076205.032702:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3190 00000100:00080000:3.0:1769076205.032717:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076205.032729:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076205.032732:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076205.032772:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076205.032782:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616519424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076205.032808:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076205.032820:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076205.032829:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076205.032832:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076205.032839:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3190 00000100:00080000:3.0:1769076205.032845:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076205.032851:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076205.032863:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076205.032890:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616519424:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076205.032895:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616519552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076205.032930:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076205.032937:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076205.032942:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076205.032955:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3190 00000100:00080000:3.0:1769076205.032966:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076205.032976:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076205.032988:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076205.033017:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616519680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076205.033043:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076205.033046:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076205.033053:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076205.033061:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076205.033072:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3190 00000100:00080000:3.0:1769076205.033083:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076205.033092:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076205.033104:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076205.033127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076205/real 1769076205] req@ffff995b8455b480 x1855007616519424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076205.033136:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616519808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076205.033191:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076205.033202:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076205.033205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616519424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076205.033209:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076205.033230:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3190 00000100:00080000:3.0:1769076205.033247:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076205.033255:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076205.033263:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769076205.033275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769076205.033280:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076205.033290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616519424:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076205.033317:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616519936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076205.033341:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076205.033345:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519552:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076205.033351:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076205.033356:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076205.033377:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3190 00000100:00080000:3.0:1769076205.033389:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076205.033398:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076205.033409:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076205.033432:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076205.033442:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616520064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076205.033509:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076205.033517:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076205.033521:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076205.033532:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3190 00000100:00080000:3.0:1769076205.033541:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076205.033549:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076205/real 1769076205] req@ffff995b84559880 x1855007616519552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076205.033557:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076205.033574:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076205.033609:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616520192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076205.033621:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616519552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076205.033685:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076205.033695:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076205.033704:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076205.033723:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076205.033781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076205.033807:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076205/real 1769076205] req@ffff995b8455ad80 x1855007616519680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076205.033835:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616519680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076205.033857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076205.033864:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076205.033871:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076205.033885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519808:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076205.033929:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076205.033954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076205/real 1769076205] req@ffff995b8455a300 x1855007616519808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076205.033981:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616519808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076205.034012:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076205.034019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076205.034026:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519808:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076205.034061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519936:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076205.034106:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076205.034131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076205/real 1769076205] req@ffff995b8455b100 x1855007616519936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076205.034157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616519936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076205.034176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076205.034183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076205.034190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616519936:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076205.034202:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616520064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076205.034240:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076205.034265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076205/real 1769076205] req@ffff995b84559c00 x1855007616520064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076205.034290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616520064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076205.034307:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076205.034314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076205.034322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616520064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076205.034334:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616520192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076205.034374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076205.034398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076205/real 1769076205] req@ffff995b84558700 x1855007616520192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076205.034423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616520192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076205.034441:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076205.034447:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076205.034509:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616520192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076205.736540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076205.736716:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076206.056657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076206.056727:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076206.056809:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076206.056882:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076206.760597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076206.760701:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076207.080583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076207.080657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076207.080699:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076207.080706:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076207.784565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076207.784733:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076208.104690:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076208.104691:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076208.104694:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076208.104709:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076208.808622:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076208.808704:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076209.128599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076209.128635:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076209.128650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076209.128672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076209.832580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076209.832677:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076210.152609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076210.152628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076210.152645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076210.152655:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076210.152666:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076210.152672:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076210.152680:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076210.152687:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3195 00000100:00080000:3.0:1769076210.152704:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076210.152714:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076210.152761:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616520320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076210.152797:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076210.152802:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076210.152805:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076210.152810:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3195 00000100:00080000:3.0:1769076210.152814:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076210.152819:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076210.152830:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076210.152846:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616520448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076210.152861:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076210.152865:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076210.152867:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076210.152876:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3195 00000100:00100000:1.0:1769076210.152878:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616520320:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076210.152880:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076210.152888:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076210.152897:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076210.152909:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616520576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076210.152921:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076210.152925:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076210.152928:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076210.152931:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3195 00000100:00080000:3.0:1769076210.152938:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076210.152945:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076210.152956:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076210.152976:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616520704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076210.152995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076210.152997:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076210.153007:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076210.153013:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076210.153024:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3195 00000100:00080000:3.0:1769076210.153033:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076210.153041:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076210.153051:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076210.153072:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076210/real 1769076210] req@ffff995b84558000 x1855007616520320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076210.153076:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616520832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076210.153109:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076210.153115:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076210.153119:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076210.153121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616520320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076210.153129:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3195 00000100:00080000:3.0:1769076210.153140:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076210.153151:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076210.153160:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076210.153181:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616520960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076210.153185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076210.153201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076210.153206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076210.153208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769076210.153221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616520320:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076210.153223:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076210.153238:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3195 00000100:00080000:3.0:1769076210.153243:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076210.153248:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076210.153254:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076210.153277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616520448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076210.153277:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616521088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076210.153341:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076210.153367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076210/real 1769076210] req@ffff995b8455bb80 x1855007616520448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076210.153392:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616520448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076210.153409:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076210.153416:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076210.153424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616520448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076210.153436:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616520576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076210.153518:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076210.153540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076210/real 1769076210] req@ffff995b8455aa00 x1855007616520576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076210.153596:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616520576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076210.153616:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076210.153623:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076210.153630:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616520576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076210.153643:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616520704:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076210.153684:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076210.153707:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076210/real 1769076210] req@ffff995b84559500 x1855007616520704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076210.153732:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616520704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076210.153748:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076210.153754:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076210.153760:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616520704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076210.153772:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616520832:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076210.153806:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076210.153827:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076210/real 1769076210] req@ffff995b84558a80 x1855007616520832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076210.153851:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616520832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076210.153867:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076210.153873:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076210.153879:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616520832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076210.153892:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616520960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076210.153925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076210.153944:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076210/real 1769076210] req@ffff995b8455a680 x1855007616520960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076210.153965:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616520960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076210.153980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076210.153996:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076210.154002:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616520960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076210.154037:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616521088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076210.154073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076210.154094:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076210/real 1769076210] req@ffff995b84558000 x1855007616521088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076210.154117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616521088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076210.154132:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076210.154137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076210.154144:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616521088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076210.856621:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076210.856682:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076211.176623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076211.176634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076211.176676:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076211.176730:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076211.880593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076211.880676:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076212.200577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076212.200595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076212.200617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076212.200666:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076212.904625:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076212.904727:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076213.224631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076213.224649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076213.224670:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076213.224703:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076213.928569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076213.928640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076214.248581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076214.248653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076214.248658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076214.248665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076214.952603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076214.952698:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076215.272617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076215.272642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076215.272664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076215.272682:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076215.272693:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076215.272700:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076215.272719:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3200 00000100:00100000:0.0:1769076215.272733:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076215.272738:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076215.272768:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076215.272823:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616521216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076215.272893:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076215.272901:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076215.272905:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076215.272914:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3200 00000100:00080000:3.0:1769076215.272924:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076215.272935:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076215.272952:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076215.272969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616521216:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076215.272982:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616521344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076215.273001:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076215.273006:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076215.273023:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076215.273029:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3200 00000100:00080000:3.0:1769076215.273035:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076215.273040:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076215.273049:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076215.273062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076215.273072:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616521472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076215.273095:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076215.273102:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076215.273106:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076215.273115:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3200 00000100:00080000:3.0:1769076215.273122:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076215.273137:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076215.273148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076215/real 1769076215] req@ffff995b84558700 x1855007616521216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076215.273151:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076215.273185:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616521600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076215.273198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616521216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076215.273215:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076215.273228:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076215.273233:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076215.273247:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3200 00000100:00080000:1.0:1769076215.273249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076215.273253:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076215.273266:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769076215.273272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769076215.273277:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076215.273296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616521216:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076215.273334:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616521728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076215.273340:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616521344:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076215.273389:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076215.273396:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076215.273401:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076215.273410:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3200 00000800:00000100:1.0:1769076215.273421:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076215.273424:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076215.273437:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076215.273449:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076215.273540:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616521856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076215.273541:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076215/real 1769076215] req@ffff995b84559c00 x1855007616521344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076215.273591:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076215.273601:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076215.273606:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076215.273610:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616521344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076215.273621:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3200 00000100:00080000:3.0:1769076215.273640:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076215.273642:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076215.273655:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076215.273657:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076215.273667:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616521344:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076215.273681:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076215.273690:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616521472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076215.273711:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616521984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076215.273733:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076215.273756:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076215/real 1769076215] req@ffff995b8455b100 x1855007616521472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076215.273781:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616521472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076215.273794:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076215.273798:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076215.273802:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616521472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076215.273810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616521600:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076215.273831:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076215.273843:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076215/real 1769076215] req@ffff995b8455a300 x1855007616521600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076215.273875:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616521600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076215.273893:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076215.273899:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076215.273906:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616521600:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076215.273939:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616521728:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076215.273980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076215.273999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076215/real 1769076215] req@ffff995b8455ad80 x1855007616521728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076215.274014:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616521728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076215.274034:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076215.274038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076215.274042:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616521728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076215.274050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616521856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076215.274081:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076215.274102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076215/real 1769076215] req@ffff995b84559880 x1855007616521856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076215.274128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616521856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076215.274157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076215.274164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076215.274171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616521856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076215.274179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616521984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076215.274208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076215.274223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076215/real 1769076215] req@ffff995b8455b480 x1855007616521984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076215.274238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616521984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076215.274249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076215.274253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076215.274258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616521984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076215.976530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076215.976573:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076216.296542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076216.296599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076216.296610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076216.296612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076217.000578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076217.000671:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076217.320565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076217.320581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076217.320596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076217.320628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076218.024602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076218.024711:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076218.344611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076218.344673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076218.344720:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076218.344758:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076219.048554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076219.048636:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076219.368586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076219.368587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076219.368589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076219.368649:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076220.072516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076220.072598:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076220.392577:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076220.392578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076220.392578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076220.392588:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076220.392592:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076220.392592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076220.392606:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3206 00000100:00080000:2.0:1769076220.392617:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076220.392623:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076220.392645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076220.392659:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616522112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076220.392687:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076220.392691:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076220.392693:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076220.392696:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3206 00000100:00080000:2.0:1769076220.392699:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076220.392702:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076220.392710:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076220.392720:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616522240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076220.392723:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616522112:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076220.392737:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076220.392740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076220.392741:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076220.392744:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3206 00000100:00080000:2.0:1769076220.392746:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076220.392748:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076220.392752:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076220.392764:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616522368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076220.392775:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076220.392778:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076220.392781:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076220.392784:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3206 00000800:00000100:1.0:1769076220.392789:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076220.392790:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076220.392798:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076220.392805:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076220.392821:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616522496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076220.392828:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076220/real 1769076220] req@ffff995b85acbb80 x1855007616522112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076220.392831:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076220.392838:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076220.392842:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076220.392847:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3206 00000100:00080000:2.0:1769076220.392851:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076220.392853:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616522112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076220.392857:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076220.392868:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769076220.392870:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076220.392876:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076220.392883:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616522112:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076220.392883:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616522624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076220.392894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076220.392896:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076220.392897:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076220.392900:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3206 00000100:00080000:2.0:1769076220.392903:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076220.392906:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616522240:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076220.392908:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076220.392916:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076220.392930:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616522752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076220.392933:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076220.392943:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076220/real 1769076220] req@ffff995b85ac8380 x1855007616522240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076220.392964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076220.392966:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616522240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076220.392974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076220.392979:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076220.392979:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076220.392987:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076220.392989:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3206 00000100:00100000:1.0:1769076220.392994:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616522240:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076220.392996:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076220.393001:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616522368:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076220.393003:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076220.393011:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076220.393019:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076220.393026:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616522880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076220.393030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076220/real 1769076220] req@ffff995b85acb100 x1855007616522368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076220.393048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616522368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076220.393055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076220.393057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076220.393060:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616522368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076220.393065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616522496:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076220.393098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076220.393105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076220/real 1769076220] req@ffff995b85ac8a80 x1855007616522496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076220.393113:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616522496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076220.393118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076220.393120:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076220.393123:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616522496:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076220.393127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616522624:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076220.393136:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076220.393142:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076220/real 1769076220] req@ffff995b85aca300 x1855007616522624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076220.393149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616522624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076220.393154:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076220.393156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076220.393158:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616522624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076220.393180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616522752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076220.393189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076220.393195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076220/real 1769076220] req@ffff995b85acb800 x1855007616522752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076220.393209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616522752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076220.393215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076220.393217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076220.393220:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616522752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076220.393223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616522880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076220.393233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076220.393240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076220/real 1769076220] req@ffff995b85ac9c00 x1855007616522880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076220.393248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616522880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076220.393253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076220.393255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076220.393256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616522880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076221.096512:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076221.096567:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076221.416594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076221.416617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076221.416648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076221.416650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076222.120564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076222.120709:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076222.440589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076222.440630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076222.440660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076222.440686:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076223.144630:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076223.144679:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076223.464570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076223.464600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076223.464636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076223.464647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076224.168603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076224.168662:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076224.488582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076224.488622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076224.488694:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076224.488764:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076225.192587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076225.192677:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076225.512598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076225.512628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076225.512630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076225.512655:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076225.512698:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076225.512707:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076225.512713:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076225.512727:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3211 00000100:00080000:3.0:1769076225.512742:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076225.512752:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076225.512817:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616523008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076225.512853:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076225.512858:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076225.512861:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076225.512866:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3211 00000100:00080000:3.0:1769076225.512871:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076225.512875:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076225.512886:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076225.512913:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616523136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076225.512930:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076225.512932:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616523008:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076225.512941:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076225.512947:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076225.512954:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3211 00000100:00080000:3.0:1769076225.512958:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076225.512962:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076225.512970:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076225.512985:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616523264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076225.512999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076225.513003:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076225.513005:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076225.513009:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3211 00000100:00080000:3.0:1769076225.513014:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076225.513019:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076225.513024:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769076225.513034:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076225.513042:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616523392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076225.513058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076225.513063:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076225.513066:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076225.513070:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3211 00000100:00080000:3.0:1769076225.513074:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076225.513079:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076225.513086:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076225.513097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076225/real 1769076225] req@ffff995b8455b480 x1855007616523008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076225.513106:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616523520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076225.513151:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076225.513156:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076225.513159:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076225.513168:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3211 00000100:00100000:1.0:1769076225.513169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616523008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076225.513173:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076225.513186:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076225.513197:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769076225.513208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076225.513229:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616523648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076225.513241:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076225.513242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076225.513249:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076225.513255:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076225.513259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616523008:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076225.513266:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3211 00000100:00080000:3.0:1769076225.513279:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076225.513332:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616523136:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076225.513349:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076225.513363:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076225.513377:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076225.513397:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616523776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076225.513404:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076225/real 1769076225] req@ffff995b84559880 x1855007616523136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076225.513499:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616523136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076225.513512:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076225.513518:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076225.513523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616523136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076225.513532:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616523264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076225.513560:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076225.513577:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076225/real 1769076225] req@ffff995b8455ad80 x1855007616523264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076225.513593:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616523264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076225.513605:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076225.513609:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076225.513613:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616523264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076225.513620:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616523392:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076225.513640:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076225.513650:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076225/real 1769076225] req@ffff995b8455a300 x1855007616523392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076225.513665:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616523392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076225.513676:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076225.513680:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076225.513683:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616523392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076225.513690:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616523520:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076225.513708:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076225.513719:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076225/real 1769076225] req@ffff995b8455b100 x1855007616523520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076225.513732:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616523520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076225.513743:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076225.513747:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076225.513750:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616523520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076225.513757:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616523648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076225.513774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076225.513785:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076225/real 1769076225] req@ffff995b84559c00 x1855007616523648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076225.513799:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616523648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076225.513817:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076225.513821:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076225.513824:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616523648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076225.513847:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616523776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076225.513865:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076225.513877:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076225/real 1769076225] req@ffff995b84558700 x1855007616523776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076225.513891:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616523776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076225.513901:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076225.513905:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076225.513909:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616523776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076226.216540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076226.216734:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076226.536744:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076226.536752:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076226.536883:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076226.536927:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076227.240606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076227.240648:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076227.560610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076227.560634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076227.560656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076227.560678:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076228.264686:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076228.264765:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076228.584571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076228.584573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076228.584581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076228.584587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076229.288573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076229.288694:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076229.608608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076229.608611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076229.608619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076229.608691:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076230.312552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076230.312715:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076230.632590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076230.632591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076230.632597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076230.632610:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076230.632621:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076230.632630:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076230.632645:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3216 00000100:00080000:3.0:1769076230.632662:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076230.632676:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076230.632677:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076230.632738:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616523904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076230.632779:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076230.632787:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076230.632791:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076230.632799:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3216 00000100:00080000:3.0:1769076230.632807:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076230.632816:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076230.632832:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076230.632854:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616523904:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076230.632856:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616524032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076230.632876:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076230.632882:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076230.632886:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076230.632892:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3216 00000100:00080000:3.0:1769076230.632898:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076230.632905:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076230.632913:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076230.632931:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616524160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076230.632945:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076230.632959:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076230.632965:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076230.632969:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076230.632975:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3216 00000100:00080000:3.0:1769076230.632981:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076230.632987:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076230.632996:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076230.633002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076230/real 1769076230] req@ffff995b84558000 x1855007616523904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076230.633013:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616524288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076230.633029:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076230.633036:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616523904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076230.633036:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076230.633040:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076230.633047:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3216 00000100:00080000:3.0:1769076230.633053:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076230.633057:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076230.633059:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769076230.633065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769076230.633067:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076230.633073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616523904:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076230.633082:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616524416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076230.633097:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076230.633102:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076230.633106:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076230.633109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524032:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076230.633114:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3216 00000100:00080000:3.0:1769076230.633121:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076230.633128:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076230.633137:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076230.633146:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076230.633154:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616524544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076230.633169:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076230/real 1769076230] req@ffff995b8455a680 x1855007616524032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076230.633172:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076230.633179:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076230.633183:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076230.633190:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3216 00000100:00100000:1.0:1769076230.633194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616524032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076230.633197:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076230.633241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076230.633248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076230.633252:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076230.633255:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524032:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076230.633261:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076230.633267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076230.633281:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616524672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076230.633320:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076230.633344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076230/real 1769076230] req@ffff995b84558a80 x1855007616524160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076230.633370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616524160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076230.633394:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076230.633398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076230.633403:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076230.633411:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524288:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076230.633440:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076230.633483:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076230/real 1769076230] req@ffff995b84559500 x1855007616524288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076230.633501:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616524288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076230.633513:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076230.633517:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076230.633521:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076230.633529:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524416:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076230.633551:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076230.633564:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076230/real 1769076230] req@ffff995b8455aa00 x1855007616524416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076230.633578:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616524416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076230.633588:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076230.633592:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076230.633596:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076230.633618:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616524544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076230.633637:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076230.633648:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076230/real 1769076230] req@ffff995b8455bb80 x1855007616524544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076230.633662:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616524544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076230.633672:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076230.633675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076230.633679:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616524544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076230.633686:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616524672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076230.633702:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076230.633714:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076230/real 1769076230] req@ffff995b84558000 x1855007616524672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076230.633728:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616524672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076230.633737:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076230.633740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076230.633744:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616524672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076231.336576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076231.336684:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076231.656516:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076231.656535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076231.656569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076231.656614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076232.360593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076232.360643:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076232.680551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076232.680556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076232.680574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076232.680620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076233.384543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076233.384675:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076233.704595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076233.704634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076233.704639:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076233.704679:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076234.408598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076234.408675:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076234.728637:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076234.728637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076234.728663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076234.728698:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076235.432547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076235.432738:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076235.752612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076235.752656:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076235.752699:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076235.752707:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076235.752739:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076235.752749:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076235.752755:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076235.752768:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3221 00000100:00080000:1.0:1769076235.752782:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076235.752792:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076235.752837:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616524800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076235.752872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076235.752878:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076235.752880:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076235.752897:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3221 00000100:00080000:1.0:1769076235.752902:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076235.752908:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076235.752920:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076235.752939:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616524800:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076235.752941:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616524928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076235.752959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076235.752971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076235.752974:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076235.752978:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3221 00000100:00080000:1.0:1769076235.752982:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076235.752987:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076235.752993:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076235.753008:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616525056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076235.753020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076235.753025:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076235.753027:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076235.753031:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3221 00000100:00080000:1.0:1769076235.753035:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076235.753040:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076235.753048:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076235.753063:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076235.753068:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616525184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076235.753085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076235.753090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076235.753104:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076235.753112:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3221 00000100:00080000:1.0:1769076235.753141:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076235.753148:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076235.753155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076235/real 1769076235] req@ffff995b868c4a80 x1855007616524800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076235.753160:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076235.753189:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616525312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076235.753198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616524800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076235.753219:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076235.753236:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076235.753242:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076235.753258:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3221 00000100:00080000:3.0:1769076235.753263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076235.753264:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076235.753278:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076235.753282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769076235.753286:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076235.753294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616524800:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076235.753309:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616525440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076235.753324:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076235.753329:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076235.753335:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076235.753336:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524928:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076235.753345:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3221 00000100:00080000:1.0:1769076235.753352:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076235.753357:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076235.753364:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076235.753383:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076235.753393:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616525568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076235.753407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076235/real 1769076235] req@ffff995b868c7800 x1855007616524928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076235.753441:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616524928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076235.753484:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076235.753491:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076235.753495:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616524928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076235.753505:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525056:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076235.753531:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076235.753545:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076235/real 1769076235] req@ffff995b868c5500 x1855007616525056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076235.753560:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616525056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076235.753572:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076235.753577:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076235.753581:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076235.753598:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525184:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076235.753617:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076235.753628:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076235/real 1769076235] req@ffff995b868c5c00 x1855007616525184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076235.753642:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616525184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076235.753653:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076235.753656:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076235.753660:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076235.753666:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525312:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076235.753683:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076235.753694:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076235/real 1769076235] req@ffff995b868c6d80 x1855007616525312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076235.753708:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616525312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076235.753719:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076235.753722:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076235.753726:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076235.753732:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616525440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076235.753750:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076235.753761:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076235/real 1769076235] req@ffff995b868c6300 x1855007616525440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076235.753776:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616525440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076235.753786:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076235.753790:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076235.753793:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616525440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076235.753815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616525568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076235.753833:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076235.753845:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076235/real 1769076235] req@ffff995b868c7100 x1855007616525568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076235.753870:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616525568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076235.753879:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076235.753883:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076235.753886:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616525568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076236.456572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076236.456732:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076236.776584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076236.776609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076236.776628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076236.776656:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076237.480593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076237.480647:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076237.800611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076237.800611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076237.800654:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076237.800691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076238.504570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076238.504722:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076238.824564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076238.824620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076238.824622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076238.824663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076239.528569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076239.528684:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076239.848527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076239.848553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076239.848559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076239.848571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076240.552553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076240.552663:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076240.872551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076240.872574:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076240.872574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076240.872582:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076240.872587:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076240.872598:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3226 00000100:00100000:2.0:1769076240.872602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076240.872611:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076240.872622:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076240.872637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076240.872664:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616525696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076240.872688:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076240.872691:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076240.872694:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076240.872697:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3226 00000100:00080000:1.0:1769076240.872701:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076240.872706:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076240.872714:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076240.872716:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616525696:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076240.872732:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616525824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076240.872743:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076240.872746:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076240.872748:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076240.872753:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3226 00000100:00080000:1.0:1769076240.872758:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076240.872762:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076240.872768:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076240.872798:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616525952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076240.872810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076240.872815:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076240.872821:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076240.872823:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076240.872832:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3226 00000100:00080000:1.0:1769076240.872841:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076240.872847:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076240.872855:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076240.872876:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616526080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076240.872886:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076240/real 1769076240] req@ffff995b868c7480 x1855007616525696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076240.872901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076240.872911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076240.872916:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076240.872919:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616525696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076240.872930:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3226 00000100:00080000:0.0:1769076240.872942:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076240.872944:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076240.872952:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076240.872960:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076240.872965:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616525696:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076240.872977:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076240.873017:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616526208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076240.873025:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525824:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076240.873046:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076240.873050:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076240.873052:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076240.873057:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3226 00000100:00080000:1.0:1769076240.873062:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076240.873066:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076240.873071:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076240.873078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076240.873085:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616526336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076240.873095:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076240.873098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076240.873101:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076240.873102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076240/real 1769076240] req@ffff995b868c6a00 x1855007616525824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076240.873106:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3226 00000100:00080000:1.0:1769076240.873112:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076240.873119:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076240.873125:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076240.873133:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616525824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076240.873137:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616526464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076240.873155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076240.873163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076240.873168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076240.873175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076240.873199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076240.873216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076240/real 1769076240] req@ffff995b868c4700 x1855007616525952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076240.873231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616525952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076240.873239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076240.873243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076240.873246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616525952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076240.873251:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526080:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076240.873265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076240.873276:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076240/real 1769076240] req@ffff995b868c7100 x1855007616526080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076240.873291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616526080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076240.873302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076240.873306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076240.873328:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076240.873362:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526208:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076240.873408:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076240.873421:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076240/real 1769076240] req@ffff995b868c6300 x1855007616526208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076240.873435:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616526208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076240.873445:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076240.873448:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076240.873452:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526208:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076240.873458:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616526336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076240.873474:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076240.873535:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076240/real 1769076240] req@ffff995b868c6d80 x1855007616526336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076240.873545:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616526336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076240.873559:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076240.873563:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076240.873566:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616526336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076240.873573:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616526464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076240.873595:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076240.873616:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076240/real 1769076240] req@ffff995b868c5c00 x1855007616526464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076240.873630:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616526464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076240.873641:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076240.873645:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076240.873649:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616526464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076241.576532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076241.576683:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076241.896595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076241.896605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076241.896657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076241.896670:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076242.600538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076242.600616:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076242.920560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076242.920596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076242.920620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076242.920651:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076243.624570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076243.624708:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076243.944578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076243.944606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076243.944619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076243.944633:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076244.648581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076244.648751:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076244.968598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076244.968600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076244.968610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076244.968666:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076245.672573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076245.672739:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076245.992623:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076245.992634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076245.992638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076245.992643:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076245.992656:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076245.992675:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3231 00000100:00100000:3.0:1769076245.992677:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076245.992695:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076245.992701:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076245.992711:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076245.992764:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616526592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076245.992824:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076245.992829:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076245.992832:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076245.992839:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3231 00000100:00080000:2.0:1769076245.992844:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076245.992851:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076245.992863:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076245.992884:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616526592:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076245.992886:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616526720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076245.992911:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076245.992916:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076245.992918:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076245.992922:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3231 00000100:00080000:2.0:1769076245.992936:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076245.992965:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076245.992975:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076245.992994:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616526848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076245.993015:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076245.993019:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076245.993022:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076245.993027:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3231 00000100:00080000:2.0:1769076245.993033:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076245.993038:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076245.993045:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076245.993052:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076245.993065:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616526976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076245.993080:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076245.993085:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076245.993087:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076245.993091:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3231 00000100:00080000:2.0:1769076245.993096:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076245.993101:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076245.993108:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076245.993119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076245/real 1769076245] req@ffff995b85ac9880 x1855007616526592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076245.993127:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616527104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076245.993171:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076245.993184:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076245.993191:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076245.993199:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3231 00000100:00080000:2.0:1769076245.993206:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076245.993217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616526592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076245.993218:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076245.993231:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076245.993250:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076245.993254:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616527232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076245.993261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076245.993275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616526592:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076245.993286:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076245.993293:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076245.993296:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076245.993303:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3231 00000100:00080000:2.0:1769076245.993306:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076245.993311:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076245.993318:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076245.993320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076245.993336:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616527360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076245.993366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076245.993394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076245/real 1769076245] req@ffff995b85acaa00 x1855007616526720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076245.993420:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616526720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076245.993436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076245.993443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076245.993450:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076245.993513:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076245.993548:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076245.993565:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076245/real 1769076245] req@ffff995b85ac9c00 x1855007616526848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076245.993588:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616526848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076245.993618:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076245.993625:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076245.993632:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076245.993645:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076245.993686:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076245.993707:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076245/real 1769076245] req@ffff995b85acb800 x1855007616526976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076245.993724:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616526976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076245.993742:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076245.993748:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076245.993755:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616526976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076245.993766:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616527104:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076245.993806:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076245.993829:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076245/real 1769076245] req@ffff995b85aca300 x1855007616527104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076245.993853:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616527104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076245.993865:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076245.993869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076245.993874:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616527104:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076245.993882:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616527232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076245.993921:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076245.993965:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076245/real 1769076245] req@ffff995b85ac8a80 x1855007616527232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076245.993995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616527232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076245.994013:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076245.994019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076245.994026:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616527232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076245.994061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616527360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076245.994092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076245.994107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076245/real 1769076245] req@ffff995b85acb100 x1855007616527360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076245.994122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616527360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076245.994133:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076245.994136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076245.994140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616527360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076246.696616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076246.696747:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076247.016581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076247.016625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076247.016678:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076247.016756:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076247.720546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076247.720626:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076248.040698:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076248.040698:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076248.040732:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076248.040814:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076248.744561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076248.744661:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076249.064607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076249.064614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076249.064698:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076249.064706:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076249.768603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076249.768652:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076250.088639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076250.088658:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076250.088689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076250.088720:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076250.792572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076250.792718:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076251.112631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076251.112636:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076251.112639:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076251.112643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076251.112649:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076251.112659:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076251.112677:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3236 00000100:00100000:2.0:1769076251.112691:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076251.112694:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076251.112735:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076251.112801:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616527488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076251.112837:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076251.112844:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076251.112848:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076251.112881:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3236 00000100:00080000:1.0:1769076251.112889:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076251.112906:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076251.112920:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076251.112946:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616527616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076251.112959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616527488:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076251.112967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076251.112972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076251.112976:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076251.112981:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3236 00000100:00080000:1.0:1769076251.112987:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076251.112993:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076251.113000:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076251.113016:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616527744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076251.113030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076251.113035:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076251.113038:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076251.113044:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3236 00000100:00080000:1.0:1769076251.113049:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076251.113050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076251.113055:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076251.113062:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076251.113079:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616527872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076251.113093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076251.113099:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076251.113102:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076251.113108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076251/real 1769076251] req@ffff995b868c5500 x1855007616527488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076251.113109:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3236 00000100:00080000:1.0:1769076251.113115:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076251.113121:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076251.113129:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076251.113138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616527488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076251.113144:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616528000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076251.113157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076251.113161:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076251.113164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076251.113166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076251.113170:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076251.113171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616527488:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076251.113176:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3236 00000100:00080000:1.0:1769076251.113181:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076251.113187:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076251.113194:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076251.113217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616527616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076251.113220:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616528128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076251.113237:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076251.113243:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076251.113246:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076251.113253:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3236 00000800:00000100:0.0:1769076251.113254:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076251.113258:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076251.113265:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076251.113273:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076251.113275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076251/real 1769076251] req@ffff995b868c7800 x1855007616527616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076251.113289:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616528256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076251.113298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616527616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076251.113314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076251.113320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076251.113326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616527616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076251.113338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616527744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076251.113374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076251.113393:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076251/real 1769076251] req@ffff995b868c4a80 x1855007616527744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076251.113408:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616527744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076251.113420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076251.113423:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076251.113427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616527744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076251.113443:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616527872:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076251.113499:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076251.113512:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076251/real 1769076251] req@ffff995b868c5c00 x1855007616527872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076251.113526:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616527872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076251.113537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076251.113540:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076251.113544:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616527872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076251.113551:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528000:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076251.113568:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076251.113579:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076251/real 1769076251] req@ffff995b868c6d80 x1855007616528000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076251.113593:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616528000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076251.113603:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076251.113606:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076251.113609:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076251.113632:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616528128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076251.113649:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076251.113660:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076251/real 1769076251] req@ffff995b868c6300 x1855007616528128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076251.113673:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616528128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076251.113683:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076251.113686:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076251.113689:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616528128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076251.113695:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616528256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076251.113710:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076251.113720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076251/real 1769076251] req@ffff995b868c7100 x1855007616528256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076251.113732:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616528256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076251.113750:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076251.113754:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076251.113757:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616528256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076251.816615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076251.816712:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076252.136627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076252.136638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076252.136644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076252.136696:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076252.840581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076252.840720:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076253.160613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076253.160670:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076253.160682:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076253.160705:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076253.864631:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076253.864723:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076254.184653:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076254.184654:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076254.184716:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076254.184723:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076254.888586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076254.888689:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076255.208599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076255.208632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076255.208664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076255.208716:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076255.912618:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076255.912726:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076256.232596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076256.232641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076256.232665:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076256.232669:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076256.232684:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076256.232694:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076256.232697:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076256.232713:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3241 00000100:00080000:3.0:1769076256.232735:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076256.232749:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076256.232831:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616528384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076256.232868:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076256.232873:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076256.232876:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076256.232881:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3241 00000100:00080000:3.0:1769076256.232887:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076256.232894:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076256.232906:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076256.232931:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616528512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076256.232933:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616528384:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076256.232962:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076256.232967:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076256.232970:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076256.232976:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3241 00000100:00080000:3.0:1769076256.232981:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076256.232985:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076256.232992:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076256.233009:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616528640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076256.233022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076256.233027:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076256.233029:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076256.233036:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3241 00000800:00000100:0.0:1769076256.233045:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076256.233047:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076256.233056:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076256.233064:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076256.233079:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616528768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076256.233092:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076256.233097:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076256.233105:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076256.233106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076256/real 1769076256] req@ffff995b84558700 x1855007616528384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076256.233121:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3241 00000100:00080000:3.0:1769076256.233138:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076256.233149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616528384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076256.233151:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076256.233171:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769076256.233174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076256.233187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076256.233193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616528384:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076256.233208:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616528896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076256.233228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528512:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076256.233239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076256.233248:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076256.233253:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076256.233260:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3241 00000800:00000100:0.0:1769076256.233268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076256.233273:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076256.233285:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076256.233294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076256/real 1769076256] req@ffff995b84559c00 x1855007616528512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076256.233295:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076256.233327:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616529024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076256.233329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616528512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076256.233358:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076256.233362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076256.233369:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528512:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076256.233390:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076256.233396:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528640:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076256.233401:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076256.233407:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076256.233416:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3241 00000100:00080000:3.0:1769076256.233423:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076256.233429:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769076256.233439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076256.233444:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076256.233496:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616529152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076256.233518:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076256/real 1769076256] req@ffff995b8455b100 x1855007616528640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076256.233559:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616528640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076256.233581:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076256.233588:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076256.233593:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076256.233602:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528768:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076256.233635:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076256.233650:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076256/real 1769076256] req@ffff995b8455a300 x1855007616528768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076256.233665:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616528768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076256.233685:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076256.233689:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076256.233693:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076256.233714:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076256.233732:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076256.233743:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076256/real 1769076256] req@ffff995b8455ad80 x1855007616528896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076256.233757:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616528896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076256.233768:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076256.233798:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076256.233802:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616528896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076256.233810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616529024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076256.233842:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076256.233855:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076256/real 1769076256] req@ffff995b84559880 x1855007616529024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076256.233870:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616529024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076256.233881:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076256.233884:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076256.233889:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616529024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076256.233896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616529152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076256.233913:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076256.233924:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076256/real 1769076256] req@ffff995b8455b480 x1855007616529152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076256.233938:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616529152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076256.233948:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076256.233951:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076256.233955:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616529152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076256.936570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076256.936636:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076257.256611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076257.256620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076257.256701:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076257.256707:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076257.960601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076257.960651:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076258.280621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076258.280627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076258.280676:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076258.280686:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076258.984601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076258.984710:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076259.304625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076259.304652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076259.304658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076259.304693:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076260.008586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076260.008769:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076260.328660:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076260.328747:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076260.328751:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076260.328838:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076261.032531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076261.032776:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076261.352590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076261.352605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076261.352621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076261.352626:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076261.352639:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076261.352647:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076261.352662:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3247 00000100:00100000:2.0:1769076261.352675:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076261.352681:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076261.352718:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076261.352782:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616529280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076261.352821:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076261.352827:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076261.352829:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076261.352835:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3247 00000100:00080000:3.0:1769076261.352840:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076261.352846:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076261.352859:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076261.352877:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616529408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076261.352885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616529280:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076261.352906:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076261.352911:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076261.352914:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076261.352921:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3247 00000100:00080000:3.0:1769076261.352926:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076261.352930:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076261.352936:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076261.352953:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616529536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076261.352966:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076261.352971:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076261.352973:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076261.352977:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3247 00000100:00080000:3.0:1769076261.352981:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076261.352986:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076261.352991:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076261.353005:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616529664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076261.353019:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769076261.353020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076261.353029:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076261.353034:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076261.353041:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3247 00000100:00080000:3.0:1769076261.353047:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076261.353053:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076261.353059:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076261.353073:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616529792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076261.353087:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076261.353094:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076261/real 1769076261] req@ffff995b8455b480 x1855007616529280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076261.353095:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076261.353102:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076261.353117:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3247 00000100:00080000:3.0:1769076261.353139:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076261.353142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616529280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076261.353150:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076261.353163:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076261.353184:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616529920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076261.353185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076261.353201:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076261.353208:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076261.353216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616529280:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076261.353217:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076261.353224:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076261.353235:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3247 00000100:00080000:3.0:1769076261.353239:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076261.353244:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076261.353250:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076261.353267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616529408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076261.353269:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616530048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076261.353318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076261.353339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076261/real 1769076261] req@ffff995b84559880 x1855007616529408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076261.353355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616529408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076261.353366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076261.353370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076261.353374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616529408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076261.353382:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616529536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076261.353400:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076261.353412:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076261/real 1769076261] req@ffff995b8455ad80 x1855007616529536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076261.353425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616529536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076261.353438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076261.353442:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076261.353446:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616529536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076261.353454:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616529664:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076261.353512:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076261.353526:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076261/real 1769076261] req@ffff995b8455a300 x1855007616529664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076261.353540:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616529664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076261.353553:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076261.353557:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076261.353561:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616529664:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076261.353568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616529792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076261.353588:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076261.353598:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076261/real 1769076261] req@ffff995b8455b100 x1855007616529792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076261.353612:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616529792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076261.353623:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076261.353627:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076261.353630:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616529792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076261.353637:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616529920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076261.353655:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076261.353667:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076261/real 1769076261] req@ffff995b84559c00 x1855007616529920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076261.353682:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616529920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076261.353719:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076261.353723:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076261.353727:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616529920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076261.353753:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616530048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076261.353777:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076261.353790:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076261/real 1769076261] req@ffff995b84558700 x1855007616530048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076261.353805:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616530048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076261.353815:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076261.353819:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076261.353823:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616530048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076262.056582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076262.056657:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076262.376565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076262.376584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076262.376592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076262.376641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076263.080612:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076263.080741:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076263.400581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076263.400588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076263.400642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076263.400712:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076264.104522:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076264.104631:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076264.424552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076264.424579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076264.424606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076264.424667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076265.128676:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076265.128715:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076265.448631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076265.448661:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076265.448661:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076265.448739:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076266.152679:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076266.152703:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076266.472667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076266.472711:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076266.472732:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076266.472746:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769076266.472750:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076266.472755:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076266.472755:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076266.472785:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3252 00000100:00080000:0.0:1769076266.472803:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076266.472814:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076266.472864:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616530176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076266.472904:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076266.472910:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076266.472912:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076266.472918:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3252 00000100:00080000:0.0:1769076266.472922:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076266.472926:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076266.472939:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076266.472955:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616530304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076266.472970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076266.472975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076266.472980:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076266.472982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616530176:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076266.472990:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3252 00000100:00080000:0.0:1769076266.472999:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076266.473003:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076266.473010:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076266.473026:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616530432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076266.473047:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076266.473052:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076266.473054:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076266.473059:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3252 00000100:00080000:0.0:1769076266.473065:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076266.473069:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076266.473074:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076266.473087:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616530560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076266.473098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076266.473103:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076266.473106:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076266.473110:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3252 00000100:00080000:0.0:1769076266.473116:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076266.473120:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076266.473125:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076266.473144:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616530688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076266.473159:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076266.473167:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616530304:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076266.473168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076266.473174:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076266.473187:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3252 00000100:00080000:0.0:1769076266.473195:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076266.473204:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076266.473212:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076266.473234:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616530432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076266.473237:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616530816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076266.473255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076266.473260:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076266.473263:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076266.473269:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3252 00000100:00080000:0.0:1769076266.473277:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076266.473284:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616530560:192.168.204.132@tcp:8:kworker.0 00000100:00080000:0.0:1769076266.473285:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076266.473302:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076266.473322:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616530944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076266.473339:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616530688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076266.473374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616530816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076266.473399:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616530944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076267.176605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076267.176756:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076267.496642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076267.496650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076267.496771:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076267.496801:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076268.200597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076268.200740:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076268.520641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076268.520642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076268.520671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076268.520698:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076269.224588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076269.224725:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076269.544628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076269.544714:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076269.544791:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076269.544908:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:1.0:1769076269.545083:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076269.545174:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076269.545239:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076269.545310:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076269.545325:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076269.545351:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076269.545373:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076269.545387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076266/real 1769076269] req@ffff995b84640e00 x1855007616530304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1769076269.545401:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076269.545426:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076269.545464:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616530304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076269.545579:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076269.545595:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076269.545607:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616530304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076269.545623:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076266/real 1769076269] req@ffff995b84642a00 x1855007616530432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076269.545674:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616530432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076269.545710:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076269.545720:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076269.545729:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616530432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076269.545747:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076266/real 1769076269] req@ffff995b84643800 x1855007616530560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076269.545781:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616530560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076269.545802:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076269.545810:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076269.545817:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616530560:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076269.545834:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076266/real 1769076269] req@ffff995b84641180 x1855007616530176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076269.545866:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616530176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076269.545882:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076269.545889:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076269.545896:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616530176:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076269.546397:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076266/real 1769076269] req@ffff995b84641500 x1855007616530944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076269.546430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616530944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076269.546448:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076269.546492:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076269.546500:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616530944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076269.546515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076266/real 1769076269] req@ffff995b84643b80 x1855007616530688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076269.546733:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616530688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076269.546756:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076269.546765:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076269.546775:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616530688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076269.546792:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076266/real 1769076269] req@ffff995b84641880 x1855007616530816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076269.546822:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616530816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076269.546965:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076269.546973:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076269.546979:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616530816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076270.248644:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076270.248697:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076270.568614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076270.568647:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076270.568688:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076270.568726:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076271.272608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076271.272701:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076271.592621:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076271.592633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076271.592635:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076271.592646:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076271.592648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076271.592663:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3257 00000100:00080000:1.0:1769076271.592679:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076271.592680:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076271.592694:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076271.592696:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076271.592745:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616531072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076271.592781:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076271.592786:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076271.592788:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076271.592794:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3257 00000100:00080000:1.0:1769076271.592798:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076271.592803:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076271.592814:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076271.592830:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616531200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076271.592848:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616531072:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076271.592852:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076271.592866:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076271.592870:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076271.592878:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3257 00000100:00080000:1.0:1769076271.592885:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076271.592892:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076271.592902:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076271.592927:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616531328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076271.592946:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076271.592953:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076271.592957:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076271.592969:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3257 00000800:00000100:3.0:1769076271.592972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076271.592980:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076271.592992:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076271.593002:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076271.593029:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616531456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076271.593048:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076271.593055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076271/real 1769076271] req@ffff995b868c4700 x1855007616531072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076271.593057:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076271.593065:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076271.593082:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3257 00000100:00080000:1.0:1769076271.593093:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076271.593101:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076271.593111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616531072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076271.593115:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769076271.593144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076271.593151:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616531584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076271.593159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076271.593177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616531072:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076271.593193:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076271.593204:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076271.593208:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076271.593218:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3257 00000100:00080000:1.0:1769076271.593225:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076271.593233:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616531200:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076271.593242:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076271.593256:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076271.593283:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616531712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076271.593305:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076271.593307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076271.593319:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076271.593323:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076271.593334:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3257 00000100:00080000:1.0:1769076271.593342:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076271.593346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076271/real 1769076271] req@ffff995b868c6a00 x1855007616531200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076271.593356:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076271.593374:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076271.593393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616531200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076271.593405:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616531840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076271.593424:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076271.593442:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076271.593491:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616531200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076271.593510:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616531328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076271.593581:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076271.594344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076271/real 1769076271] req@ffff995b868c7480 x1855007616531328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076271.594376:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616531328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076271.594394:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076271.594400:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076271.594406:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616531328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076271.594419:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616531456:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076271.594501:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076271.594553:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076271/real 1769076271] req@ffff995b868c7100 x1855007616531456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076271.594574:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616531456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076271.594589:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076271.594596:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076271.594602:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616531456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076271.594613:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616531584:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076271.594652:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076271.594674:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076271/real 1769076271] req@ffff995b868c6300 x1855007616531584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076271.594707:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616531584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076271.594724:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076271.594729:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076271.594735:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616531584:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076271.594772:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616531712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076271.594825:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076271.594849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076271/real 1769076271] req@ffff995b868c6d80 x1855007616531712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076271.594873:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616531712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076271.594888:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076271.594895:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076271.594901:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616531712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076271.594913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616531840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076271.594948:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076271.594968:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076271/real 1769076271] req@ffff995b868c5c00 x1855007616531840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076271.594985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616531840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076271.595004:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076271.595008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076271.595011:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616531840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076272.296671:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076272.296806:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076272.616638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076272.616734:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076272.616785:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076272.616793:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076273.320577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076273.320709:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076273.640633:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076273.640682:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076273.640685:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076273.640722:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076274.344531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076274.344624:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076274.664557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076274.664583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076274.664584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076274.664627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076275.368530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076275.368570:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076275.688556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076275.688574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076275.688588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076275.688613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076276.392530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076276.392607:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076276.712531:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076276.712570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076276.712579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076276.712590:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076276.712598:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076276.712603:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076276.712613:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3262 00000100:00080000:1.0:1769076276.712625:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076276.712634:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076276.712662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076276.712686:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616531968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076276.712728:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076276.712735:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076276.712739:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076276.712746:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3262 00000100:00080000:1.0:1769076276.712753:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076276.712760:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076276.712772:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076276.712783:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616531968:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076276.712796:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616532096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076276.712811:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076276.712815:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076276.712818:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076276.712824:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3262 00000100:00080000:1.0:1769076276.712830:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076276.712835:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076276.712843:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076276.712857:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616532224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076276.712869:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769076276.712870:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076276.712877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076276.712880:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076276.712887:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3262 00000100:00080000:1.0:1769076276.712892:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076276.712897:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076276.712901:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076276.712933:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616532352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076276.712944:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076276.712951:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769076276.712951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076276/real 1769076276] req@ffff995b868c4a80 x1855007616531968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076276.712957:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076276.712970:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3262 00000100:00080000:1.0:1769076276.712998:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076276.713005:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076276.713010:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076276.713022:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616531968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076276.713026:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616532480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076276.713068:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076276.713069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076276.713075:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076276.713081:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076276.713081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076276.713090:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616531968:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076276.713091:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3262 00000100:00080000:1.0:1769076276.713101:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076276.713105:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076276.713109:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076276.713117:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616532608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076276.713125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532096:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076276.713129:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076276.713134:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076276.713136:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076276.713142:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3262 00000100:00080000:1.0:1769076276.713146:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076276.713149:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076276.713155:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076276.713161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076276.713170:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616532736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076276.713182:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076276/real 1769076276] req@ffff995b868c7800 x1855007616532096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076276.713211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616532096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076276.713219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076276.713222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076276.713227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076276.713235:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076276.713254:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076276.713266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076276/real 1769076276] req@ffff995b868c5500 x1855007616532224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076276.713278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616532224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076276.713289:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076276.713295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076276.713302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076276.713315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532352:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076276.713351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076276.713394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076276/real 1769076276] req@ffff995b868c5c00 x1855007616532352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076276.713409:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616532352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076276.713418:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076276.713422:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076276.713449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076276.713484:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532480:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076276.713510:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076276.713528:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076276/real 1769076276] req@ffff995b868c6d80 x1855007616532480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076276.713546:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616532480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076276.713560:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076276.713565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076276.713570:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076276.713595:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616532608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076276.713625:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076276.713638:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076276/real 1769076276] req@ffff995b868c6300 x1855007616532608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076276.713650:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616532608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076276.713658:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076276.713661:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076276.713664:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616532608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076276.713669:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616532736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076276.713683:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076276.713691:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076276/real 1769076276] req@ffff995b868c7100 x1855007616532736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076276.713701:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616532736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076276.713710:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076276.713712:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076276.713715:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616532736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076277.416527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076277.416584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076277.736534:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076277.736546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076277.736562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076277.736579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076278.440571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076278.440685:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076278.760619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076278.760869:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076278.760894:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076278.761022:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076279.464553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076279.464715:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076279.784567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076279.784621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076279.784639:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076279.784642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076280.488586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076280.488784:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076280.808656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076280.808660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076280.808703:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076280.808723:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076281.512538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076281.512579:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076281.832517:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076281.832517:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076281.832537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076281.832548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076281.832549:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076281.832555:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076281.832559:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076281.832567:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3267 00000100:00080000:3.0:1769076281.832576:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076281.832583:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076281.832612:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616532864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076281.832633:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076281.832636:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076281.832638:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076281.832642:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3267 00000100:00080000:3.0:1769076281.832644:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076281.832648:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076281.832655:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076281.832665:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616532864:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076281.832666:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616532992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076281.832679:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076281.832681:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076281.832682:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076281.832684:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3267 00000100:00080000:3.0:1769076281.832686:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076281.832689:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076281.832692:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076281.832700:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616533120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076281.832707:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076281.832709:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076281.832711:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076281.832713:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3267 00000100:00080000:3.0:1769076281.832717:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076281.832719:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076281.832722:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076281.832729:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616533248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076281.832735:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076281.832737:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076281.832740:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076281.832741:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076281.832745:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3267 00000100:00080000:3.0:1769076281.832748:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076281.832751:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076281.832754:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076281.832761:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616533376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076281.832767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076281.832769:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076281.832770:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076281.832773:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3267 00000100:00080000:3.0:1769076281.832776:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076281.832780:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076281/real 1769076281] req@ffff995b84558000 x1855007616532864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076281.832782:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076281.832788:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076281.832800:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616533504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076281.832805:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616532864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076281.832808:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076281.832810:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076281.832811:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076281.832819:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3267 00000100:00080000:3.0:1769076281.832827:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076281.832830:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076281.832833:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076281.832837:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769076281.832842:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076281.832843:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616532864:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076281.832862:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616533632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076281.832868:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076281.832893:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076281.832904:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076281/real 1769076281] req@ffff995b8455bb80 x1855007616532992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076281.832912:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616532992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076281.832918:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076281.832921:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076281.832923:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616532992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076281.832927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616533120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076281.832937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076281.832943:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076281/real 1769076281] req@ffff995b8455aa00 x1855007616533120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076281.832951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616533120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076281.832957:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076281.832959:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076281.832961:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616533120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076281.832965:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616533248:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076281.832975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076281.832980:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076281/real 1769076281] req@ffff995b84559500 x1855007616533248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076281.832993:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616533248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076281.833001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076281.833004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076281.833007:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616533248:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076281.833013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616533376:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076281.833031:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076281.833041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076281/real 1769076281] req@ffff995b84558a80 x1855007616533376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076281.833049:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616533376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076281.833056:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076281.833058:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076281.833060:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616533376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076281.833064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616533504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076281.833076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076281.833082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076281/real 1769076281] req@ffff995b8455a680 x1855007616533504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076281.833093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616533504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076281.833108:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076281.833112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076281.833115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616533504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076281.833137:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616533632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076281.833154:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076281.833162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076281/real 1769076281] req@ffff995b84558700 x1855007616533632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076281.833169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616533632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076281.833175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076281.833177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076281.833180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616533632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076282.536546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076282.536632:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076282.856518:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076282.856534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076282.856562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076282.856563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076283.560586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076283.560677:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076283.880644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076283.880660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076283.880683:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076283.880710:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076284.584568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076284.584653:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076284.904612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076284.904625:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076284.904675:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076284.904685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076285.608579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076285.608662:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076285.928586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076285.928613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076285.928644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076285.928694:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076286.632637:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076286.632733:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076286.952589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076286.952616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076286.952648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076286.952709:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076286.952726:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076286.952736:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076286.952743:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076286.952756:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3272 00000100:00080000:1.0:1769076286.952773:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076286.952785:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076286.952853:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616533760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076286.952894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076286.952901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076286.952905:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076286.952914:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3272 00000100:00080000:1.0:1769076286.952923:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076286.952932:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076286.952950:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076286.952961:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616533760:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076286.952986:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616533888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076286.953007:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076286.953012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076286.953016:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076286.953022:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3272 00000100:00080000:1.0:1769076286.953029:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076286.953034:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076286.953042:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076286.953061:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616534016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076286.953073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076286.953086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076286.953092:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076286.953095:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076286.953103:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3272 00000100:00080000:1.0:1769076286.953108:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076286.953112:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076286.953119:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076286.953133:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616534144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076286.953147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076286.953155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769076286.953156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076286/real 1769076286] req@ffff995b868c7480 x1855007616533760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076286.953159:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076286.953175:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3272 00000100:00080000:1.0:1769076286.953185:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076286.953195:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076286.953203:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076286.953211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616533760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076286.953220:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616534272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076286.953246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076286.953250:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076286.953291:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076286.953295:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076286.953297:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076286.953306:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3272 00000100:00100000:2.0:1769076286.953330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616533760:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076286.953333:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076286.953348:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076286.953355:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076286.953371:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616534400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076286.953376:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616533888:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076286.953402:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076286.953407:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076286.953411:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076286.953418:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3272 00000100:00080000:1.0:1769076286.953424:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076286.953433:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076286.953434:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076286.953446:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076286.953502:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076286/real 1769076286] req@ffff995b868c6a00 x1855007616533888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076286.953518:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616534528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076286.953528:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616533888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076286.953552:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076286.953556:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076286.953562:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616533888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076286.953573:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076286.953597:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076286.953612:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076286/real 1769076286] req@ffff995b868c4700 x1855007616534016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076286.953627:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616534016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076286.953639:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076286.953643:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076286.953647:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076286.953654:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534144:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076286.953673:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076286.953683:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076286/real 1769076286] req@ffff995b868c7100 x1855007616534144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076286.953696:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616534144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076286.953706:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076286.953709:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076286.953713:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534144:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076286.953719:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076286.953736:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076286.953753:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076286/real 1769076286] req@ffff995b868c6300 x1855007616534272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076286.953774:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616534272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076286.953791:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076286.953798:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076286.953805:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076286.953837:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616534400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076286.953876:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076286.953896:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076286/real 1769076286] req@ffff995b868c6d80 x1855007616534400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076286.953911:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616534400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076286.953921:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076286.953925:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076286.953928:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616534400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076286.953935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616534528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076286.953956:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076286.953967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076286/real 1769076286] req@ffff995b868c5c00 x1855007616534528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076286.953980:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616534528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076286.953989:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076286.953992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076286.953997:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616534528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076287.656558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076287.656672:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076287.976594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076287.976627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076287.976627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076287.976671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076288.680579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076288.680712:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076289.000640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076289.000666:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076289.000709:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076289.000721:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076289.704653:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076289.704733:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076290.024613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076290.024613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076290.024621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076290.024729:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076290.728585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076290.728690:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076291.048740:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076291.048749:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076291.048779:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076291.048797:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076291.752532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076291.752574:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076292.072529:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076292.072532:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076292.072547:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076292.072553:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769076292.072556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076292.072557:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076292.072568:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3277 00000100:00100000:2.0:1769076292.072572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076292.072579:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076292.072587:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076292.072618:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616534656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076292.072637:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076292.072640:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076292.072641:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076292.072651:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3277 00000100:00080000:3.0:1769076292.072656:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076292.072660:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076292.072668:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076292.072671:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616534656:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076292.072680:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616534784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076292.072690:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076292.072692:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076292.072694:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076292.072696:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3277 00000100:00080000:3.0:1769076292.072699:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076292.072702:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076292.072705:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076292.072713:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616534912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076292.072720:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076292.072723:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076292.072725:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076292.072727:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3277 00000100:00080000:3.0:1769076292.072729:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076292.072732:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076292.072736:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076292.072738:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076292.072749:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616535040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076292.072760:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076292.072763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076292.072764:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076292.072768:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3277 00000100:00080000:3.0:1769076292.072771:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076292.072774:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076292/real 1769076292] req@ffff995b84559c00 x1855007616534656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076292.072776:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076292.072783:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076292.072797:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616534656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076292.072801:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616535168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076292.072810:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076292.072813:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076292.072817:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076292.072818:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076292.072821:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076292.072822:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616534656:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076292.072828:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3277 00000100:00080000:3.0:1769076292.072831:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076292.072835:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076292.072838:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076292.072846:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076292.072848:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616535296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076292.072857:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076292.072860:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076292.072862:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076292.072866:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076292.072867:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3277 00000100:00080000:3.0:1769076292.072873:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076292.072876:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076292.072880:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076292.072881:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076292/real 1769076292] req@ffff995b8455b100 x1855007616534784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076292.072892:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616535424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076292.072904:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616534784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076292.072912:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076292.072914:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076292.072917:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076292.072922:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076292.072934:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076292.072940:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076292/real 1769076292] req@ffff995b8455a300 x1855007616534912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076292.072948:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616534912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076292.072955:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076292.072956:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076292.072958:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616534912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076292.072962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535040:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076292.072971:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076292.072977:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076292/real 1769076292] req@ffff995b8455ad80 x1855007616535040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076292.072984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616535040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076292.072995:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076292.072997:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076292.072999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076292.073002:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535168:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076292.073020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076292.073025:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076292/real 1769076292] req@ffff995b84559880 x1855007616535168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076292.073032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616535168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076292.073038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076292.073040:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076292.073042:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076292.073051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616535296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076292.073061:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076292.073067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076292/real 1769076292] req@ffff995b8455b480 x1855007616535296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076292.073074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616535296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076292.073079:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076292.073081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076292.073084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616535296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076292.073087:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616535424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076292.073096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076292.073102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076292/real 1769076292] req@ffff995b84559c00 x1855007616535424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076292.073109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616535424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076292.073114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076292.073116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076292.073117:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616535424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076292.776571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076292.776686:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076293.096630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076293.096688:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076293.096708:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076293.096737:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076293.800560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076293.800663:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076294.120579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076294.120594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076294.120614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076294.120664:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076294.824597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076294.824705:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076295.144573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076295.144630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076295.144705:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076295.144707:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076295.848611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076295.848732:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076296.168574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076296.168628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076296.168657:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076296.168673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076296.872535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076296.872655:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076297.192557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076297.192633:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076297.192633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076297.192653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076297.192670:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076297.192685:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076297.192694:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076297.192710:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3282 00000100:00080000:0.0:1769076297.192729:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076297.192760:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076297.192816:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616535552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076297.192855:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076297.192860:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076297.192863:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076297.192869:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3282 00000100:00080000:0.0:1769076297.192875:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076297.192882:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076297.192895:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076297.192926:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616535680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076297.192928:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616535552:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076297.192959:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076297.192964:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076297.192967:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076297.192974:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3282 00000100:00080000:0.0:1769076297.192979:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076297.192984:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076297.192990:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076297.193009:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616535808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076297.193021:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076297.193026:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076297.193028:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076297.193034:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3282 00000100:00080000:0.0:1769076297.193039:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076297.193043:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076297.193051:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076297.193056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076297.193071:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616535936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076297.193111:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076297.193115:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076297.193118:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076297.193123:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3282 00000100:00080000:0.0:1769076297.193138:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076297.193144:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076297.193151:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076297.193170:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616536064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076297.193183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076297/real 1769076297] req@ffff995b84642d80 x1855007616535552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076297.193188:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076297.193202:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076297.193207:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076297.193223:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3282 00000100:00080000:0.0:1769076297.193230:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076297.193235:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076297.193242:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076297.193249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616535552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076297.193263:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616536192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076297.193276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076297.193282:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076297.193290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076297.193292:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076297.193301:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076297.193307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616535552:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076297.193314:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3282 00000100:00080000:0.0:1769076297.193324:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076297.193330:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076297.193337:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076297.193349:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616536320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076297.193359:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076297.193413:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076297.193443:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076297/real 1769076297] req@ffff995b84643100 x1855007616535680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076297.193540:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616535680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076297.193560:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076297.193569:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076297.193577:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076297.193591:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076297.193636:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076297.193662:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076297/real 1769076297] req@ffff995b84643480 x1855007616535808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076297.193688:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616535808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076297.193710:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076297.193714:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076297.193719:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076297.193728:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535936:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076297.193758:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076297.193772:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076297/real 1769076297] req@ffff995b84641880 x1855007616535936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076297.193786:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616535936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076297.193797:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076297.193801:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076297.193805:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616535936:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076297.193812:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536064:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076297.193830:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076297.193841:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076297/real 1769076297] req@ffff995b84643b80 x1855007616536064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076297.193854:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616536064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076297.193865:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076297.193869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076297.193874:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536064:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076297.193880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616536192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076297.193910:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076297.193930:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076297/real 1769076297] req@ffff995b84641500 x1855007616536192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076297.193955:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616536192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076297.193972:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076297.193979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076297.193986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616536192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076297.194024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616536320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076297.194059:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076297.194110:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076297/real 1769076297] req@ffff995b84641180 x1855007616536320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076297.194138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616536320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076297.194157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076297.194164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076297.194171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616536320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076297.896550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076297.896634:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076298.216542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076298.216563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076298.216586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076298.216636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076298.920547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076298.920610:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076299.240581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076299.240590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076299.240605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076299.240644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076299.944595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076299.944676:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076300.264581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076300.264593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076300.264634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076300.264654:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076300.968605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076300.968658:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076301.288562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076301.288578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076301.288606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076301.288632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076301.992594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076301.992670:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076302.312607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076302.312608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076302.312641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076302.312694:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076302.312731:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076302.312741:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076302.312746:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076302.312758:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3288 00000100:00080000:0.0:1769076302.312772:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076302.312781:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076302.312827:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616536448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076302.312865:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076302.312870:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076302.312873:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076302.312878:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3288 00000100:00080000:0.0:1769076302.312884:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076302.312890:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076302.312903:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076302.312927:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616536576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076302.312929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616536448:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076302.312963:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076302.312969:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076302.312971:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076302.312977:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3288 00000100:00080000:0.0:1769076302.312982:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076302.312986:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076302.312992:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076302.313042:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616536704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076302.313055:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076302.313059:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076302.313061:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076302.313067:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3288 00000100:00080000:0.0:1769076302.313072:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076302.313077:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076302.313087:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076302.313088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076302.313113:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616536832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076302.313135:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076302.313140:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076302.313143:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076302.313153:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3288 00000100:00100000:2.0:1769076302.313155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076302/real 1769076302] req@ffff995b84643800 x1855007616536448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076302.313158:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076302.313170:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076302.313181:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076302.313203:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616536960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076302.313210:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616536448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076302.313226:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076302.313241:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076302.313247:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076302.313252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076302.313256:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3288 00000100:00080000:0.0:1769076302.313263:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076302.313265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076302.313274:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076302.313281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616536448:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076302.313285:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076302.313305:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616537088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076302.313321:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076302.313331:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076302.313331:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536576:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076302.313337:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076302.313347:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3288 00000100:00080000:0.0:1769076302.313354:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076302.313359:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076302.313366:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076302.313381:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616537216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076302.313397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076302.313438:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076302/real 1769076302] req@ffff995b84642a00 x1855007616536576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076302.313521:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616536576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076302.313534:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076302.313538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076302.313543:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076302.313553:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076302.313585:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076302.313600:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076302/real 1769076302] req@ffff995b84640e00 x1855007616536704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076302.313615:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616536704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076302.313627:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076302.313630:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076302.313634:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076302.313641:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536832:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076302.313660:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076302.313671:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076302/real 1769076302] req@ffff995b84640000 x1855007616536832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076302.313685:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616536832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076302.313696:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076302.313699:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076302.313703:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076302.313710:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536960:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076302.313728:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076302.313739:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076302/real 1769076302] req@ffff995b84641c00 x1855007616536960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076302.313754:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616536960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076302.313765:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076302.313768:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076302.313772:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616536960:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076302.313796:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616537088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076302.313814:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076302.313826:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076302/real 1769076302] req@ffff995b84642680 x1855007616537088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076302.313839:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616537088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076302.313849:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076302.313853:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076302.313857:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616537088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076302.313863:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616537216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076302.313880:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076302.313891:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076302/real 1769076302] req@ffff995b84640700 x1855007616537216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076302.313904:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616537216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076302.313923:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076302.313927:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076302.313930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616537216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076303.016586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076303.016732:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076303.336577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076303.336612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076303.336621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076303.336635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076304.040567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076304.040722:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076304.360555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076304.360595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076304.360660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076304.360684:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076305.064584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076305.064670:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076305.384626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076305.384736:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076305.384752:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076305.384757:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076306.088588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076306.088690:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076306.408631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076306.408659:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076306.408705:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076306.408720:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076307.112584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076307.112689:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076307.432621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076307.432642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076307.432658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076307.432733:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076307.432735:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076307.432749:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076307.432756:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076307.432773:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3293 00000100:00080000:2.0:1769076307.432788:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076307.432800:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076307.432850:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616537344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076307.432890:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076307.432895:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076307.432898:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076307.432903:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3293 00000100:00080000:2.0:1769076307.432909:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076307.432948:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076307.432961:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076307.432976:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616537344:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076307.432987:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616537472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076307.433005:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076307.433010:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076307.433015:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076307.433023:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3293 00000100:00080000:2.0:1769076307.433031:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076307.433039:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076307.433048:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076307.433076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076307.433080:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616537600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076307.433107:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076307.433114:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076307.433119:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076307.433129:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3293 00000100:00080000:2.0:1769076307.433138:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076307.433138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076307/real 1769076307] req@ffff995b85ac8380 x1855007616537344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076307.433154:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076307.433173:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076307.433182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616537344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076307.433197:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616537728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076307.433203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076307.433217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076307.433223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616537344:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076307.433253:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076307.433258:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769076307.433263:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616537472:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076307.433265:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076307.433282:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3293 00000100:00080000:2.0:1769076307.433290:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076307.433295:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076307.433304:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076307.433314:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076307.433339:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616537856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076307.433340:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076307/real 1769076307] req@ffff995b85acbb80 x1855007616537472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076307.433419:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076307.433428:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616537472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076307.433430:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076307.433437:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076307.433487:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3293 00000100:00080000:2.0:1769076307.433496:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076307.433503:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076307.433511:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076307.433533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076307.433544:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616537984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076307.433551:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076307.433568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616537472:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076307.433577:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076307.433584:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076307.433586:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616537600:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076307.433589:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076307.433607:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3293 00000100:00080000:2.0:1769076307.433613:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076307.433618:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076307.433627:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076307.433649:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076307.433651:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616538112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076307.433676:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076307/real 1769076307] req@ffff995b85acb100 x1855007616537600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076307.433695:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616537600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076307.433709:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076307.433714:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076307.433719:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616537600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076307.433727:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616537728:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076307.433749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076307.433761:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076307/real 1769076307] req@ffff995b85ac8a80 x1855007616537728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076307.433784:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616537728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076307.433795:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076307.433799:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076307.433802:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616537728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076307.433825:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616537856:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076307.433844:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076307.433855:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076307/real 1769076307] req@ffff995b85aca300 x1855007616537856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076307.433869:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616537856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076307.433880:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076307.433884:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076307.433887:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616537856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076307.433894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616537984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076307.433942:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076307.433960:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076307/real 1769076307] req@ffff995b85acb800 x1855007616537984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076307.433975:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616537984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076307.433994:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076307.433998:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076307.434002:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616537984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076307.434009:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616538112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076307.434033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076307.434044:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076307/real 1769076307] req@ffff995b85ac9c00 x1855007616538112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076307.434058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616538112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076307.434068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076307.434072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076307.434076:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616538112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076308.136625:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076308.136729:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076308.456606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076308.456610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076308.456674:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076308.456768:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076309.160639:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076309.160708:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076309.480539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076309.480545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076309.480552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076309.480579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076310.184518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076310.184569:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076310.504529:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076310.504550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076310.504561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076310.504577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076311.208563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076311.208690:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076311.528594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076311.528596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076311.528650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076311.528766:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076312.232593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076312.232695:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076312.552592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076312.552611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076312.552637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076312.552698:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076312.552760:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076312.552773:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076312.552780:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076312.552796:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3298 00000100:00080000:0.0:1769076312.552813:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076312.552851:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076312.552909:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616538240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076312.552972:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076312.552981:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076312.552987:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076312.553000:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3298 00000100:00080000:0.0:1769076312.553012:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076312.553025:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076312.553043:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076312.553048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616538240:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076312.553080:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616538368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076312.553101:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076312.553105:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076312.553108:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076312.553114:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3298 00000100:00080000:0.0:1769076312.553120:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076312.553125:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076312.553133:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076312.553183:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616538496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076312.553200:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076312.553213:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076312.553217:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076312.553225:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3298 00000100:00080000:0.0:1769076312.553232:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076312.553239:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076312.553247:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076312.553266:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616538624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076312.553281:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076312.553288:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076312.553293:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769076312.553297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076312.553306:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3298 00000100:00080000:0.0:1769076312.553314:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076312.553320:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076312.553328:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076312.553347:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616538752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076312.553361:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076312.553368:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769076312.553373:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076312/real 1769076312] req@ffff995b84640380 x1855007616538240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076312.553374:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076312.553386:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3298 00000100:00080000:0.0:1769076312.553395:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076312.553406:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076312.553415:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076312.553427:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616538880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076312.553438:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076312.553442:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076312.553444:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076312.553448:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3298 00000100:00080000:0.0:1769076312.553488:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076312.553495:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076312.553502:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076312.553518:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616539008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076312.553666:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616538240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076312.553704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076312.553714:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076312.553811:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616538240:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076312.553870:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616538368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076312.553912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076312.553932:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076312/real 1769076312] req@ffff995b84641f80 x1855007616538368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076312.553949:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616538368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076312.553961:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076312.553966:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076312.553970:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616538368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076312.553978:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616538496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076312.553998:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076312.554010:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076312/real 1769076312] req@ffff995b84640a80 x1855007616538496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076312.554024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616538496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076312.554037:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076312.554041:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076312.554045:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616538496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076312.554053:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616538624:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076312.554072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076312.554084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076312/real 1769076312] req@ffff995b84640700 x1855007616538624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076312.554098:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616538624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076312.554109:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076312.554113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076312.554116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616538624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076312.554123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616538752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076312.554141:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076312.554152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076312/real 1769076312] req@ffff995b84642680 x1855007616538752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076312.554166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616538752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076312.554177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076312.554180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076312.554184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616538752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076312.554191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616538880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076312.554209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076312.554220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076312/real 1769076312] req@ffff995b84641c00 x1855007616538880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076312.554235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616538880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076312.554246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076312.554249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076312.554252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616538880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076312.554259:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616539008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076312.554277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076312.554289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076312/real 1769076312] req@ffff995b84640000 x1855007616539008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076312.554303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616539008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076312.554312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076312.554316:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076312.554319:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616539008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076313.256606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076313.256692:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076313.576630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076313.576651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076313.576665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076313.576699:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076314.280602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076314.280757:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076314.600658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076314.600676:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076314.600676:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076314.600680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076315.304594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076315.304732:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076315.624579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076315.624601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076315.624601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076315.624638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076316.328620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076316.328704:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076316.648592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076316.648595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076316.648608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076316.648670:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076317.352587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076317.352721:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076317.672578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076317.672579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076317.672667:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076317.672675:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076317.672682:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769076317.672683:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076317.672692:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076317.672711:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3303 00000100:00080000:3.0:1769076317.672731:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076317.672770:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076317.672828:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616539136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076317.672878:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076317.672884:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076317.672888:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076317.672895:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3303 00000100:00080000:3.0:1769076317.672902:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076317.672909:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076317.672922:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076317.672931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616539136:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076317.672950:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616539264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076317.672968:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076317.672973:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076317.672983:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076317.672989:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3303 00000100:00080000:3.0:1769076317.673005:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076317.673011:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076317.673018:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076317.673027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076317.673041:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616539392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076317.673057:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076317.673062:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076317.673065:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076317.673070:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3303 00000100:00080000:3.0:1769076317.673076:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076317.673081:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076317.673090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076317/real 1769076317] req@ffff995b84558700 x1855007616539136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076317.673091:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076317.673114:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616539520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076317.673156:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076317.673161:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769076317.673166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616539136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076317.673167:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076317.673181:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3303 00000100:00080000:3.0:1769076317.673189:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076317.673199:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076317.673203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769076317.673207:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769076317.673216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076317.673225:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616539648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076317.673228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616539136:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076317.673255:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076317.673260:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076317.673263:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076317.673271:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3303 00000100:00100000:2.0:1769076317.673279:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616539264:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076317.673285:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076317.673295:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076317.673304:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076317.673319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076317.673326:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616539776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076317.673344:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076317.673349:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076317.673351:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076317.673357:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3303 00000100:00080000:3.0:1769076317.673361:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076317.673363:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076317/real 1769076317] req@ffff995b8455a680 x1855007616539264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076317.673369:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076317.673382:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076317.673404:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616539904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076317.673410:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616539264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076317.673434:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076317.673441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076317.673447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616539264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076317.673490:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616539392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076317.673517:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076317.673532:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076317/real 1769076317] req@ffff995b84558a80 x1855007616539392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076317.673547:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616539392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076317.673568:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076317.673573:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076317.673577:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616539392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076317.673584:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616539520:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076317.673603:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076317.673614:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076317/real 1769076317] req@ffff995b84559500 x1855007616539520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076317.673628:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616539520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076317.673640:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076317.673643:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076317.673646:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616539520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076317.673657:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616539648:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076317.673674:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076317.673685:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076317/real 1769076317] req@ffff995b8455aa00 x1855007616539648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076317.673699:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616539648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076317.673709:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076317.673713:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076317.673717:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616539648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076317.673737:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616539776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076317.673787:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076317.673801:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076317/real 1769076317] req@ffff995b8455bb80 x1855007616539776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076317.673814:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616539776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076317.673824:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076317.673828:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076317.673832:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616539776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076317.673839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616539904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076317.673857:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076317.673868:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076317/real 1769076317] req@ffff995b84558000 x1855007616539904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076317.673882:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616539904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076317.673892:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076317.673895:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076317.673899:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616539904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076318.376631:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076318.376681:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076318.696592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076318.696594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076318.696643:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076318.696655:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076319.400568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076319.400699:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076319.720607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076319.720640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076319.720662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076319.720698:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076320.424589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076320.424748:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076320.744556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076320.744614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076320.744628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076320.744648:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076321.448588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076321.448757:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076321.768555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076321.768600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076321.768615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076321.768695:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076322.472524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076322.472584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076322.792603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076322.792622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076322.792622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076322.792707:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076322.792732:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076322.792743:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076322.792750:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076322.792765:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3308 00000100:00080000:2.0:1769076322.792779:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076322.792789:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076322.792834:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616540032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076322.792875:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076322.792881:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076322.792883:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076322.792889:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3308 00000100:00080000:2.0:1769076322.792893:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076322.792899:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076322.792910:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076322.792928:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616540160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076322.792938:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616540032:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076322.792954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076322.792960:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076322.792963:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076322.792970:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3308 00000100:00080000:2.0:1769076322.792976:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076322.792980:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076322.792988:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076322.793006:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616540288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076322.793020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076322.793024:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076322.793026:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076322.793033:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3308 00000800:00000100:3.0:1769076322.793042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076322.793044:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076322.793053:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076322.793060:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076322.793076:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616540416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076322.793088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076322.793093:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076322.793096:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076322.793103:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3308 00000100:00100000:3.0:1769076322.793105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076322/real 1769076322] req@ffff995b85acaa00 x1855007616540032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076322.793108:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076322.793122:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076322.793135:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076322.793153:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616540032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076322.793157:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616540544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076322.793196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076322.793199:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076322.793211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076322.793215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076322.793217:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076322.793227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616540032:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076322.793230:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3308 00000100:00080000:2.0:1769076322.793245:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076322.793254:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076322.793263:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076322.793271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616540160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076322.793292:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616540672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076322.793311:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076322.793331:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076322.793340:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076322/real 1769076322] req@ffff995b85ac9880 x1855007616540160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076322.793344:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076322.793353:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076322.793371:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3308 00000100:00100000:3.0:1769076322.793374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616540160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076322.793381:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076322.793396:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076322.793397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076322.793414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:2.0:1769076322.793416:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076322.793422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616540160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076322.793435:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616540288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076322.793445:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616540800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076322.793516:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076322.793544:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076322/real 1769076322] req@ffff995b85ac9c00 x1855007616540288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076322.793561:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616540288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076322.793572:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076322.793576:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076322.793581:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616540288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076322.793588:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616540416:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076322.793612:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076322.793623:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076322/real 1769076322] req@ffff995b85acb800 x1855007616540416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076322.793638:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616540416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076322.793650:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076322.793676:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076322.793680:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616540416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076322.793690:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616540544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076322.793711:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076322.793723:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076322/real 1769076322] req@ffff995b85aca300 x1855007616540544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076322.793737:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616540544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076322.793748:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076322.793752:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076322.793756:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616540544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076322.793776:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616540672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076322.793794:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076322.793805:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076322/real 1769076322] req@ffff995b85ac8a80 x1855007616540672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076322.793827:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616540672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076322.793837:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076322.793841:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076322.793844:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616540672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076322.793851:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616540800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076322.793868:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076322.793879:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076322/real 1769076322] req@ffff995b85acb100 x1855007616540800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076322.793893:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616540800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076322.793902:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076322.793906:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076322.793909:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616540800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076323.496569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076323.496717:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076323.816620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076323.816702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076323.816726:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076323.816768:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076324.520600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076324.520731:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076324.840606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076324.840660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076324.840764:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076324.840869:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076325.544585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076325.544762:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076325.864644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076325.864656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076325.864689:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076325.864742:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076326.568570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076326.568743:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076326.888677:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076326.888678:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076326.888783:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076326.888787:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076327.592625:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076327.592728:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076327.912616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076327.912688:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076327.912692:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076327.912697:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076327.912706:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076327.912719:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3313 00000100:00080000:3.0:1769076327.912736:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076327.912747:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076327.912751:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076327.912753:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076327.912805:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616540928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076327.912848:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076327.912854:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076327.912858:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076327.912865:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3313 00000100:00080000:3.0:1769076327.912871:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076327.912878:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076327.912890:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076327.912900:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616540928:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076327.912913:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616541056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076327.912931:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076327.912935:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076327.912937:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076327.912942:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3313 00000100:00080000:3.0:1769076327.912947:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076327.912951:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076327.912958:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076327.912972:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616541184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076327.912984:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076327.912990:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076327.912993:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076327.913000:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076327.913008:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3313 00000100:00080000:3.0:1769076327.913017:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076327.913022:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076327.913031:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076327.913048:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616541312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076327.913060:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076327.913067:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1769076327.913069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076327/real 1769076327] req@ffff995b84558000 x1855007616540928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076327.913072:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076327.913088:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3313 00000100:00080000:3.0:1769076327.913099:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076327.913111:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076327.913137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616540928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076327.913139:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769076327.913162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076327.913170:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616541440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076327.913177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076327.913191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616540928:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076327.913204:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076327.913211:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076327.913214:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076327.913220:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3313 00000100:00080000:3.0:1769076327.913226:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076327.913230:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076327.913236:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076327.913255:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616541568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076327.913268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541056:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076327.913274:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076327.913287:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076327.913290:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076327.913297:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3313 00000100:00080000:3.0:1769076327.913303:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076327.913307:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076327.913314:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076327.913330:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616541696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076327.913344:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076327.913382:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076327/real 1769076327] req@ffff995b8455bb80 x1855007616541056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076327.913402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616541056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076327.913420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076327.913428:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076327.913435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076327.913448:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076327.913536:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076327.913560:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076327/real 1769076327] req@ffff995b8455aa00 x1855007616541184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076327.913607:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616541184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076327.913629:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076327.913633:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076327.913637:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076327.913646:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541312:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076327.913675:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076327.913689:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076327/real 1769076327] req@ffff995b84559500 x1855007616541312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076327.913704:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616541312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076327.913715:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076327.913719:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076327.913723:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076327.913730:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076327.913749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076327.913760:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076327/real 1769076327] req@ffff995b84558a80 x1855007616541440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076327.913773:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616541440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076327.913784:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076327.913788:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076327.913793:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076327.913819:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616541568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076327.913838:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076327.913850:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076327/real 1769076327] req@ffff995b8455a680 x1855007616541568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076327.913864:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616541568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076327.913874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076327.913878:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076327.913881:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616541568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076327.913888:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616541696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076327.913905:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076327.913917:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076327/real 1769076327] req@ffff995b84558700 x1855007616541696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076327.913930:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616541696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076327.913940:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076327.913944:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076327.913947:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616541696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076328.616623:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076328.616735:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076328.936687:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076328.936687:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076328.936729:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076328.936849:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076329.640578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076329.640610:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076329.960657:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076329.960665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076329.960717:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076329.960792:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076330.664623:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076330.664707:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076330.984640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076330.984655:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076330.984666:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076330.984734:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076331.688580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076331.688687:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076332.008660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076332.008668:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076332.008690:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076332.008704:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076332.712626:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076332.712752:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076333.032620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076333.032650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076333.032659:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076333.032661:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076333.032675:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076333.032684:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076333.032702:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3318 00000100:00080000:3.0:1769076333.032725:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076333.032737:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076333.032747:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076333.032806:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616541824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076333.032846:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076333.032853:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076333.032855:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076333.032861:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3318 00000100:00080000:3.0:1769076333.032865:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076333.032872:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076333.032887:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076333.032904:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616541952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076333.032913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616541824:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076333.032930:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076333.032936:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076333.032938:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076333.032943:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3318 00000100:00080000:3.0:1769076333.032947:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076333.032952:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076333.032958:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076333.032972:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616542080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076333.032984:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076333.032988:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076333.032990:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076333.032995:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3318 00000100:00080000:3.0:1769076333.033000:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076333.033004:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076333.033011:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076333.033026:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616542208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076333.033039:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076333.033043:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076333.033045:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076333.033050:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3318 00000100:00080000:3.0:1769076333.033055:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076333.033059:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076333.033064:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076333.033078:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616542336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076333.033089:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076333.033094:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076333.033097:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076333.033107:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541952:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076333.033112:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3318 00000100:00080000:3.0:1769076333.033125:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076333.033134:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076333.033147:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076333.033171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076333.033183:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616542464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076333.033210:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076333.033212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542208:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076333.033246:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076333.033252:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076333.033260:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3318 00000100:00080000:3.0:1769076333.033270:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076333.033277:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076333.033288:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076333.033291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076333.033321:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616542592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076333.033343:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616542464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076333.033405:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616542592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076333.736617:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076333.736734:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076334.056622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076334.056653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076334.056663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076334.056738:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076334.760569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076334.760691:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076335.080623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076335.080627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076335.080637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076335.080690:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076335.784598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076335.784688:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076336.104567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076336.104632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076336.104709:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076336.104784:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:1.0:1769076336.104884:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076336.104923:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076336.104956:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076336.104985:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076336.104996:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076336.105005:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076336.105013:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076336.105019:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076333/real 1769076336] req@ffff995b8455b480 x1855007616541952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1769076336.105022:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076336.105034:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076336.105049:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616541952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076336.105060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076336.105064:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076336.105067:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616541952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076336.105073:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076333/real 1769076336] req@ffff995b84559880 x1855007616542080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076336.105084:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616542080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076336.105090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076336.105093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076336.105102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076336.105107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076333/real 1769076336] req@ffff995b8455ad80 x1855007616542208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076336.105115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616542208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076336.105121:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076336.105124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076336.105126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542208:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076336.105130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076333/real 1769076336] req@ffff995b8455a300 x1855007616542336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076336.105137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616542336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076336.105142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076336.105144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076336.105146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076336.105150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076333/real 1769076336] req@ffff995b84559c00 x1855007616541824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076336.105159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616541824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076336.105164:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076336.105166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076336.105168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616541824:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076336.105191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076333/real 1769076336] req@ffff995b84558700 x1855007616542592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076336.105198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616542592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076336.105203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076336.105205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076336.105207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616542592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076336.105212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076333/real 1769076336] req@ffff995b8455b100 x1855007616542464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076336.105243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616542464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076336.105256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076336.105260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076336.105265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616542464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076336.808551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076336.808609:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076337.128536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076337.128554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076337.128579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076337.128663:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076337.832592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076337.832704:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076338.152574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076338.152610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076338.152651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076338.152657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076338.152665:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076338.152676:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076338.152682:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076338.152694:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3323 00000100:00080000:1.0:1769076338.152709:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076338.152720:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076338.152764:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616542720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076338.152799:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076338.152805:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076338.152808:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076338.152816:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3323 00000100:00080000:1.0:1769076338.152822:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076338.152830:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076338.152846:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076338.152870:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616542848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076338.152893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616542720:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076338.152893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076338.152913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076338.152917:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076338.152926:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3323 00000100:00080000:1.0:1769076338.152935:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076338.152943:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076338.152953:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076338.152996:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616542976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076338.153016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076338.153023:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076338.153032:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076338.153034:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076338.153069:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3323 00000100:00080000:1.0:1769076338.153104:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076338.153114:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076338.153126:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076338.153154:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616543104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076338.153165:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076338/real 1769076338] req@ffff995b868c5500 x1855007616542720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076338.153181:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076338.153197:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076338.153204:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076338.153228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616542720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076338.153248:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3323 00000100:00080000:0.0:1769076338.153266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076338.153271:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076338.153287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076338.153289:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076338.153306:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076338.153308:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616542720:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076338.153338:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616543232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076338.153362:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076338.153364:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542848:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076338.153375:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076338.153382:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076338.153392:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3323 00000100:00080000:1.0:1769076338.153434:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076338.153442:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076338.153489:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076338.153515:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616543360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076338.153532:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076338.153539:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076338.153542:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076338.153552:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076338.153554:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3323 00000100:00080000:1.0:1769076338.153570:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076338.153579:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076338.153590:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076338.153595:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076338/real 1769076338] req@ffff995b868c7800 x1855007616542848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076338.153620:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616543488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076338.153635:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616542848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076338.153672:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076338.153700:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076338.153710:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076338.153725:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076338.153786:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076338.153812:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076338/real 1769076338] req@ffff995b868c4a80 x1855007616542976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076338.153836:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616542976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076338.153856:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076338.153873:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076338.153882:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616542976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076338.153895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616543104:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076338.153937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076338.153959:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076338/real 1769076338] req@ffff995b868c5c00 x1855007616543104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076338.153983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616543104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076338.154000:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076338.154006:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076338.154013:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616543104:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076338.154047:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616543232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076338.154084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076338.154106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076338/real 1769076338] req@ffff995b868c6d80 x1855007616543232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076338.154128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616543232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076338.154146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076338.154152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076338.154159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616543232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076338.154171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616543360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076338.154208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076338.154230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076338/real 1769076338] req@ffff995b868c6300 x1855007616543360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076338.154253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616543360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076338.154268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076338.154273:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076338.154280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616543360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076338.154291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616543488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076338.154328:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076338.154350:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076338/real 1769076338] req@ffff995b868c7100 x1855007616543488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076338.154375:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616543488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076338.154390:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076338.154419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076338.154427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616543488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076338.856572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076338.856695:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076339.176588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076339.176637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076339.176685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076339.176775:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076339.880558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076339.880681:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076340.200653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076340.200658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076340.200682:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076340.200723:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076340.904582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076340.904767:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076341.224589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076341.224595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076341.224692:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076341.224729:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076341.928613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076341.928727:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076342.248602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076342.248624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076342.248691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076342.248695:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076342.952647:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076342.952753:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076343.272634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076343.272635:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076343.272636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076343.272651:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076343.272659:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076343.272675:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3328 00000100:00080000:3.0:1769076343.272693:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076343.272708:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076343.272711:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076343.272714:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076343.272770:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616543616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076343.272803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076343.272812:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076343.272815:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076343.272822:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3328 00000100:00080000:3.0:1769076343.272826:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076343.272832:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076343.272843:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076343.272856:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616543616:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076343.272865:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616543744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076343.272884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076343.272889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076343.272891:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076343.272896:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3328 00000100:00080000:3.0:1769076343.272900:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076343.272904:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076343.272909:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076343.272924:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616543872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076343.272935:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076343.272940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076343.272942:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076343.272947:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3328 00000100:00080000:3.0:1769076343.272952:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076343.272956:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076343.272964:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076343.272980:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616544000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076343.272984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076343.273001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076343.273006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076343.273009:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076343.273014:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3328 00000100:00080000:3.0:1769076343.273020:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076343.273026:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076343.273031:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076343.273045:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616544128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076343.273056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076343.273060:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076343/real 1769076343] req@ffff995b8455a680 x1855007616543616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076343.273061:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076343.273068:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076343.273083:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3328 00000100:00080000:3.0:1769076343.273093:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076343.273104:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076343.273110:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076343.273128:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616544256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076343.273138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076343.273142:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076343.273146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616543616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076343.273146:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076343.273161:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3328 00000100:00080000:3.0:1769076343.273170:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076343.273181:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076343.273191:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076343.273198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076343.273207:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616544384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076343.273212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076343.273228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616543616:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076343.273271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616543744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076343.273364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076343.273384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076343/real 1769076343] req@ffff995b84558a80 x1855007616543744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076343.273402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616543744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076343.273414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076343.273418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076343.273423:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616543744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076343.273432:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616543872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076343.273488:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076343.273505:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076343/real 1769076343] req@ffff995b84559500 x1855007616543872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076343.273521:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616543872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076343.273534:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076343.273538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076343.273542:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616543872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076343.273550:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544000:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076343.273573:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076343.273585:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076343/real 1769076343] req@ffff995b8455aa00 x1855007616544000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076343.273599:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616544000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076343.273610:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076343.273613:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076343.273617:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076343.273624:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544128:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076343.273642:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076343.273653:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076343/real 1769076343] req@ffff995b8455bb80 x1855007616544128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076343.273667:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616544128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076343.273679:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076343.273682:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076343.273685:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076343.273692:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616544256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076343.273709:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076343.273721:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076343/real 1769076343] req@ffff995b84558000 x1855007616544256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076343.273735:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616544256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076343.273749:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076343.273755:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076343.273762:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616544256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076343.273773:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616544384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076343.273810:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076343.273832:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076343/real 1769076343] req@ffff995b8455b100 x1855007616544384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076343.273857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616544384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076343.273883:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076343.273887:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076343.273892:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616544384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076343.976608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076343.976667:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076344.296598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076344.296598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076344.296712:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076344.296738:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076345.000597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076345.000698:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076345.320639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076345.320663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076345.320726:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076345.320757:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076346.024588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076346.024637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076346.344550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076346.344632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076346.344653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076346.344723:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076347.048654:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076347.048746:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076347.368600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076347.368602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076347.368606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076347.368721:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076348.072643:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076348.072750:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076348.392587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076348.392645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076348.392659:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076348.392697:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076348.392708:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076348.392719:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076348.392732:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3334 00000100:00080000:0.0:1769076348.392748:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076348.392764:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076348.392771:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076348.392823:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616544512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076348.392894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076348.392903:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076348.392910:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076348.392922:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3334 00000100:00080000:0.0:1769076348.392936:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076348.392948:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076348.392966:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076348.392989:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616544512:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076348.393018:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616544640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076348.393041:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076348.393047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076348.393050:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076348.393059:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3334 00000100:00080000:0.0:1769076348.393068:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076348.393079:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076348.393090:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076348.393118:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616544768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076348.393121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076348.393151:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076348.393159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076348.393164:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076348.393172:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3334 00000100:00080000:0.0:1769076348.393183:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076348.393199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076348/real 1769076348] req@ffff995b84640e00 x1855007616544512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076348.393200:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076348.393256:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076348.393299:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616544896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076348.393324:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076348.393330:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616544512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076348.393332:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076348.393353:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076348.393356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076348.393363:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3334 00000100:00080000:0.0:1769076348.393373:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076348.393375:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076348.393384:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076348.393392:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616544512:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076348.393397:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076348.393424:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616545024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076348.393437:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544640:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076348.393447:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076348.393502:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076348.393507:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076348.393519:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3334 00000100:00080000:0.0:1769076348.393528:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076348.393538:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076348.393548:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769076348.393557:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076348.393571:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076348/real 1769076348] req@ffff995b84642a00 x1855007616544640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076348.393596:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616545152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076348.393600:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616544640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076348.393697:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076348.393698:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076348.393717:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076348.393724:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076348.393729:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076348.393739:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544640:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076348.393746:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3334 00000100:00080000:0.0:1769076348.393758:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076348.393761:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544768:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076348.393772:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076348.393787:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769076348.393788:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076348.393801:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076348/real 1769076348] req@ffff995b84643800 x1855007616544768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076348.393822:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616544768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076348.393826:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616545280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076348.393866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076348.393885:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076348.393904:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076348.393926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076348.393975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076348.393999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076348/real 1769076348] req@ffff995b84641180 x1855007616544896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076348.394014:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616544896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076348.394025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076348.394029:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076348.394033:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616544896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076348.394058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545024:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076348.394080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076348.394091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076348/real 1769076348] req@ffff995b84641500 x1855007616545024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076348.394106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616545024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076348.394117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076348.394120:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076348.394124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076348.394131:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616545152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076348.394149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076348.394159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076348/real 1769076348] req@ffff995b84643b80 x1855007616545152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076348.394172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616545152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076348.394183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076348.394186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076348.394189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616545152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076348.394196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616545280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076348.394212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076348.394246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076348/real 1769076348] req@ffff995b84641880 x1855007616545280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076348.394297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616545280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076348.394308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076348.394312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076348.394316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616545280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076349.096629:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076349.096763:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076349.416651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076349.416699:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076349.416747:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076349.416802:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076350.120631:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076350.120748:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076350.440590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076350.440615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076350.440664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076350.440665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076351.144560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076351.144693:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076351.464581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076351.464626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076351.464673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076351.464702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076352.168511:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076352.168621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076352.488524:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076352.488537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076352.488584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076352.488597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076353.192570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076353.192695:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076353.512576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076353.512601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076353.512614:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076353.512627:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076353.512634:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076353.512650:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3339 00000100:00100000:3.0:1769076353.512662:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076353.512666:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076353.512684:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076353.512701:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076353.512761:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616545408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076353.512798:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076353.512805:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076353.512809:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076353.512818:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3339 00000100:00080000:0.0:1769076353.512824:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076353.512831:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076353.512845:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076353.512876:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616545408:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076353.512877:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616545536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076353.512923:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076353.512927:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076353.512930:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076353.512937:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3339 00000100:00080000:0.0:1769076353.512943:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076353.512948:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076353.512954:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076353.512985:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616545664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076353.512998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076353.513003:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076353.513005:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076353.513010:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3339 00000100:00080000:0.0:1769076353.513025:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076353.513029:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076353.513035:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076353.513069:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616545792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076353.513080:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076353.513084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076353.513090:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076353.513095:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076353.513104:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3339 00000100:00080000:0.0:1769076353.513113:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076353.513117:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076353.513124:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076353.513168:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616545920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076353.513192:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076353.513214:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076353.513217:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076353.513222:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3339 00000100:00080000:0.0:1769076353.513230:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076353.513237:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076353.513245:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076353.513288:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616546048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076353.513305:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076353.513308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076353/real 1769076353] req@ffff995b84643480 x1855007616545408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076353.513311:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076353.513317:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076353.513330:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3339 00000100:00080000:0.0:1769076353.513345:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076353.513357:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076353.513373:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076353.513394:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616546176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076353.513417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616545408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076353.513448:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076353.513520:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076353.513531:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616545408:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076353.513594:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076353.513649:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076353.513679:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076353/real 1769076353] req@ffff995b84643100 x1855007616545536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076353.513709:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616545536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076353.513738:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076353.513746:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076353.513753:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076353.513767:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076353.513806:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076353.513828:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076353/real 1769076353] req@ffff995b84642d80 x1855007616545664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076353.513855:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616545664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076353.513879:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076353.513887:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076353.513894:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076353.513907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076353.513949:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076353.513973:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076353/real 1769076353] req@ffff995b84641880 x1855007616545792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076353.514002:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616545792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076353.514021:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076353.514028:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076353.514035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076353.514061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076353.514104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076353.514128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076353/real 1769076353] req@ffff995b84643b80 x1855007616545920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076353.514187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616545920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076353.514219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076353.514226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076353.514233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616545920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076353.514247:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616546048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076353.514293:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076353.514318:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076353/real 1769076353] req@ffff995b84641500 x1855007616546048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076353.514345:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616546048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076353.514378:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076353.514385:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076353.514393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616546048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076353.514405:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616546176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076353.514449:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076353.514516:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076353/real 1769076353] req@ffff995b84641180 x1855007616546176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076353.514544:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616546176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076353.514563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076353.514570:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076353.514576:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616546176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076354.216600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076354.216686:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076354.536604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076354.536605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076354.536651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076354.536710:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076355.240582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076355.240682:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076355.560658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076355.560664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076355.560728:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076355.560785:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076356.264543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076356.264618:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076356.584575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076356.584578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076356.584584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076356.584614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076357.288577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076357.288700:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076357.608626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076357.608632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076357.608659:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076357.608690:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076358.312607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076358.312680:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076358.632563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076358.632635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076358.632644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076358.632724:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076358.632729:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076358.632741:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076358.632753:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076358.632778:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3344 00000100:00080000:1.0:1769076358.632795:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076358.632808:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076358.632869:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616546304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076358.632909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076358.632914:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076358.632917:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076358.632922:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3344 00000100:00080000:1.0:1769076358.632930:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076358.632940:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076358.632957:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076358.632993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616546304:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076358.633005:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616546432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076358.633037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076358.633046:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076358.633077:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076358.633097:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3344 00000100:00080000:1.0:1769076358.633104:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076358.633111:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076358.633122:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076358.633142:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616546560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076358.633179:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076358.633184:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076358.633193:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076358.633198:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076358.633203:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3344 00000100:00080000:1.0:1769076358.633214:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076358.633224:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076358.633235:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076358.633264:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616546688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076358.633285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076358.633295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076358/real 1769076358] req@ffff995b868c4700 x1855007616546304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076358.633298:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076358.633308:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076358.633329:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3344 00000100:00080000:1.0:1769076358.633344:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076358.633363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616546304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076358.633367:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076358.633390:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076358.633420:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616546816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076358.633422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076358.633445:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076358.633461:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616546304:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076358.633499:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076358.633506:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076358.633510:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076358.633518:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3344 00000100:00080000:1.0:1769076358.633523:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076358.633528:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076358.633537:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076358.633552:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616546432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076358.633558:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616546944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076358.633597:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076358.633602:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076358.633607:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076358.633616:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076358.633619:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3344 00000100:00080000:1.0:1769076358.633637:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076358.633645:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076358/real 1769076358] req@ffff995b868c6a00 x1855007616546432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076358.633651:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076358.633682:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076358.633689:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616546432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076358.633708:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076358.633714:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076358.633721:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616546432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076358.633728:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616547072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076358.633740:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616546560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076358.633792:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076358.633811:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076358/real 1769076358] req@ffff995b868c7480 x1855007616546560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076358.633833:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616546560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076358.633846:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076358.633851:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076358.633854:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616546560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076358.633862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616546688:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076358.633880:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076358.633890:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076358/real 1769076358] req@ffff995b868c7100 x1855007616546688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076358.633902:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616546688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076358.633912:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076358.633915:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076358.633918:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616546688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076358.633925:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616546816:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076358.633940:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076358.633950:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076358/real 1769076358] req@ffff995b868c6300 x1855007616546816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076358.633979:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616546816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076358.634000:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076358.634004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076358.634008:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616546816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076358.634033:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616546944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076358.634110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076358.634128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076358/real 1769076358] req@ffff995b868c6d80 x1855007616546944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076358.634149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616546944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076358.634181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076358.634188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076358.634195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616546944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076358.634208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616547072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076358.634263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076358.634281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076358/real 1769076358] req@ffff995b868c5c00 x1855007616547072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076358.634295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616547072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076358.634307:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076358.634313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076358.634320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616547072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076359.336635:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076359.336728:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076359.656603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076359.656674:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076359.656682:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076359.656686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076360.360613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076360.360704:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076360.680625:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076360.680625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076360.680693:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076360.680695:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076361.384553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076361.384754:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076361.704600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076361.704620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076361.704624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076361.704692:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076362.408530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076362.408700:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076362.728542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076362.728549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076362.728594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076362.728600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076363.432530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076363.432675:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076363.752608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076363.752689:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076363.752704:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076363.752719:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076363.752779:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076363.752786:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076363.752791:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076363.752801:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3349 00000100:00080000:2.0:1769076363.752814:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076363.752822:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076363.752869:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616547200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076363.752916:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076363.752923:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076363.752926:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076363.752933:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3349 00000100:00080000:2.0:1769076363.752941:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076363.752949:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076363.752961:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076363.753018:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616547200:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076363.753028:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616547328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076363.753057:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076363.753065:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076363.753068:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076363.753078:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3349 00000100:00080000:2.0:1769076363.753088:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076363.753097:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076363.753109:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076363.753133:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616547456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076363.753148:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076363.753182:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076363.753190:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076363.753194:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076363.753206:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3349 00000100:00080000:2.0:1769076363.753219:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076363.753222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076363/real 1769076363] req@ffff995b85acbb80 x1855007616547200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076363.753235:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076363.753272:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076363.753276:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616547200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076363.753297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076363.753306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076363.753311:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616547584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076363.753315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616547200:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076363.753351:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076363.753356:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076363.753360:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076363.753365:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3349 00000100:00080000:2.0:1769076363.753373:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076363.753381:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616547328:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076363.753382:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076363.753402:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076363.753424:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616547712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076363.753440:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076363.753449:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076363.753455:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076363.753457:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076363.753463:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3349 00000100:00080000:2.0:1769076363.753533:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076363.753537:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076363/real 1769076363] req@ffff995b85ac8380 x1855007616547328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076363.753544:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076363.753554:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076363.753600:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616547840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076363.753608:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616547328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076363.753622:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076363.753650:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076363.753662:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076363.753663:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076363.753679:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076363.753686:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3349 00000100:00100000:0.0:1769076363.753695:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616547328:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076363.753707:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076363.753713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616547456:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076363.753727:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076363.753741:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076363.753766:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616547968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076363.753774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076363.753805:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076363/real 1769076363] req@ffff995b85acb100 x1855007616547456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076363.753830:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616547456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076363.753844:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076363.753849:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076363.753853:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616547456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076363.753863:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616547584:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076363.753885:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076363.753897:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076363/real 1769076363] req@ffff995b85ac8a80 x1855007616547584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076363.753912:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616547584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076363.753923:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076363.753927:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076363.753930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616547584:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076363.753955:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616547712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076363.754008:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076363.754027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076363/real 1769076363] req@ffff995b85aca300 x1855007616547712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076363.754042:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616547712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076363.754055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076363.754060:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076363.754064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616547712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076363.754072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616547840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076363.754092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076363.754104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076363/real 1769076363] req@ffff995b85acb800 x1855007616547840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076363.754118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616547840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076363.754129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076363.754132:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076363.754136:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616547840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076363.754144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616547968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076363.754162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076363.754173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076363/real 1769076363] req@ffff995b85ac9c00 x1855007616547968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076363.754188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616547968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076363.754198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076363.754201:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076363.754206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616547968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076364.456591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076364.456669:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076364.776583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076364.776621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076364.776665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076364.776690:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076365.480595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076365.480701:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076365.800569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076365.800601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076365.800617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076365.800671:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076366.504635:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076366.504636:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076366.824558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076366.824562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076366.824565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076366.824591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076367.528581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076367.528676:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076367.848578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076367.848594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076367.848608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076367.848684:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076368.552583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076368.552700:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076368.872579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076368.872621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076368.872680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076368.872714:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076368.872725:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076368.872730:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076368.872743:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076368.872763:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3354 00000100:00080000:2.0:1769076368.872784:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076368.872798:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076368.872856:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616548096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076368.872927:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076368.872933:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076368.872936:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076368.872943:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3354 00000100:00080000:2.0:1769076368.872951:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076368.872958:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076368.872974:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076368.872987:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616548096:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076368.873006:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616548224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076368.873035:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076368.873042:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076368.873046:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076368.873056:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3354 00000100:00080000:2.0:1769076368.873064:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076368.873073:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076368.873085:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076368.873091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076368.873121:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616548352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076368.873138:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076368.873143:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076368.873146:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076368.873153:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3354 00000100:00080000:2.0:1769076368.873159:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076368.873164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076368/real 1769076368] req@ffff995b85ac9880 x1855007616548096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076368.873171:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076368.873185:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076368.873203:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616548480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076368.873218:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076368.873225:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1769076368.873226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616548096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076368.873237:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076368.873263:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3354 00000100:00080000:0.0:1769076368.873269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076368.873276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076368.873282:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616548096:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076368.873284:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076368.873294:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076368.873300:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076368.873319:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616548608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076368.873324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616548224:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076368.873353:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076368.873359:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076368.873362:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076368.873369:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3354 00000100:00080000:2.0:1769076368.873375:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076368.873383:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076368.873388:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076368.873398:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076368.873405:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076368/real 1769076368] req@ffff995b85acaa00 x1855007616548224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076368.873419:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616548736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076368.873427:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616548224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076368.873487:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076368.873503:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076368.873505:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076368.873519:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076368.873521:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616548224:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076368.873524:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076368.873536:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3354 00000100:00100000:0.0:1769076368.873537:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616548352:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076368.873550:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076368.873560:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076368.873568:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076368.873578:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076368.873598:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076368/real 1769076368] req@ffff995b85ac9c00 x1855007616548352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076368.873602:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616548864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076368.873614:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616548352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076368.873665:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076368.873672:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076368.873677:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616548352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076368.873689:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616548480:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076368.873707:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076368.873717:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076368/real 1769076368] req@ffff995b85acb800 x1855007616548480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076368.873725:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616548480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076368.873732:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076368.873734:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076368.873736:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616548480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076368.873751:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616548608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076368.873763:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076368.873769:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076368/real 1769076368] req@ffff995b85aca300 x1855007616548608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076368.873778:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616548608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076368.873785:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076368.873787:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076368.873788:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616548608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076368.873793:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616548736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076368.873803:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076368.873810:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076368/real 1769076368] req@ffff995b85ac8a80 x1855007616548736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076368.873818:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616548736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076368.873824:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076368.873826:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076368.873827:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616548736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076368.873831:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616548864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076368.873841:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076368.873847:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076368/real 1769076368] req@ffff995b85acb100 x1855007616548864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076368.873854:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616548864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076368.873864:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076368.873867:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076368.873869:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616548864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076369.576583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076369.576693:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076369.896664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076369.896665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076369.896713:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076369.896742:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076370.600551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076370.600723:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076370.920678:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076370.920713:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076370.920737:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076370.920748:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076371.624597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076371.624723:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076371.944629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076371.944644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076371.944677:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076371.944692:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076372.648615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076372.648711:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076372.968657:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076372.968657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076372.968700:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076372.968743:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076373.672597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076373.672726:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076373.992585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076373.992638:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076373.992647:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076373.992653:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076373.992662:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076373.992676:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3359 00000100:00080000:3.0:1769076373.992693:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076373.992711:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076373.992712:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076373.992755:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076373.992780:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616548992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076373.992844:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076373.992849:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076373.992851:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076373.992857:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3359 00000100:00080000:3.0:1769076373.992862:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076373.992867:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076373.992879:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076373.992898:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616549120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076373.992915:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076373.992923:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076373.992926:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076373.992935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616548992:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076373.992938:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3359 00000100:00080000:3.0:1769076373.992949:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076373.992956:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076373.992963:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076373.992980:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616549248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076373.992999:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076373.993004:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076373.993007:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076373.993012:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3359 00000100:00080000:3.0:1769076373.993017:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076373.993021:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076373.993026:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076373.993043:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616549376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076373.993053:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076373.993068:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076373.993074:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076373.993076:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076373.993083:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3359 00000100:00080000:3.0:1769076373.993090:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076373.993095:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076373.993101:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076373.993116:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616549504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076373.993128:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076373.993137:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769076373.993140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076373/real 1769076373] req@ffff995b84558700 x1855007616548992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076373.993141:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076373.993155:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3359 00000100:00080000:3.0:1769076373.993164:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076373.993175:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076373.993188:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076373.993211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616548992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076373.993212:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616549632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076373.993274:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076373.993276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076373.993282:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076373.993287:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076373.993289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076373.993301:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3359 00000100:00100000:0.0:1769076373.993304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616548992:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076373.993310:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076373.993321:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076373.993326:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076373.993341:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616549760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076373.993349:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616549120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076373.993405:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076373.993437:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076373/real 1769076373] req@ffff995b84559c00 x1855007616549120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076373.993502:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616549120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076373.993523:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076373.993528:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076373.993533:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616549120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076373.993542:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616549248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076373.993567:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076373.993581:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076373/real 1769076373] req@ffff995b8455a300 x1855007616549248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076373.993595:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616549248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076373.993611:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076373.993616:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076373.993621:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616549248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076373.993633:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616549376:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076373.993670:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076373.993692:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076373/real 1769076373] req@ffff995b8455ad80 x1855007616549376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076373.993717:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616549376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076373.993736:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076373.993742:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076373.993750:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616549376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076373.993762:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616549504:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076373.993791:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076373.993844:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076373/real 1769076373] req@ffff995b84559880 x1855007616549504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076373.993868:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616549504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076373.993879:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076373.993883:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076373.993887:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616549504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076373.993894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616549632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076373.993921:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076373.993933:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076373/real 1769076373] req@ffff995b8455b480 x1855007616549632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076373.993947:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616549632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076373.993968:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076373.993971:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076373.993975:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616549632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076373.994002:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616549760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076373.994021:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076373.994043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076373/real 1769076373] req@ffff995b84558700 x1855007616549760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076373.994067:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616549760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076373.994083:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076373.994089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076373.994108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616549760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076374.696572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076374.696664:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076375.016615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076375.016630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076375.016642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076375.016689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076375.720586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076375.720724:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076376.040536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076376.040546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076376.040562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076376.040570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076376.744573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076376.744661:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076377.064626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076377.064650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076377.064651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076377.064770:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076377.768590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076377.768780:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076378.088551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076378.088561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076378.088582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076378.088625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076378.792557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076378.792753:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076379.112594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076379.112613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076379.112648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076379.112651:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076379.112664:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076379.112671:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076379.112685:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3364 00000100:00100000:2.0:1769076379.112694:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076379.112700:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076379.112736:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076379.112798:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616549888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076379.112836:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076379.112841:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076379.112844:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076379.112849:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3364 00000100:00080000:3.0:1769076379.112854:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076379.112859:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076379.112871:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076379.112888:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616550016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076379.112901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616549888:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076379.112911:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076379.112918:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076379.112921:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076379.112927:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3364 00000100:00080000:3.0:1769076379.112933:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076379.112937:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076379.112948:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076379.112966:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616550144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076379.112980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076379.112984:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076379.112986:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076379.112990:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3364 00000100:00080000:3.0:1769076379.112994:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076379.112999:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076379.113021:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076379.113028:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076379.113046:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616550272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076379.113060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076379.113064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076379.113066:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076379.113071:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3364 00000100:00080000:3.0:1769076379.113076:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076379.113085:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076379.113092:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076379.113107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076379/real 1769076379] req@ffff995b8455b100 x1855007616549888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076379.113112:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616550400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076379.113177:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076379.113183:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076379.113186:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076379.113192:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3364 00000100:00080000:3.0:1769076379.113197:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076379.113206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616549888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076379.113209:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076379.113228:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076379.113249:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616550528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076379.113251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076379.113269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076379.113277:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076379.113285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616549888:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076379.113304:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076379.113310:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076379.113317:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3364 00000100:00080000:3.0:1769076379.113322:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076379.113326:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076379.113332:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076379.113341:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076379.113352:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616550656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076379.113399:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076379.113420:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076379/real 1769076379] req@ffff995b84558000 x1855007616550016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076379.113436:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616550016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076379.113447:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076379.113452:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076379.113502:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076379.113511:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076379.113536:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076379.113550:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076379/real 1769076379] req@ffff995b8455bb80 x1855007616550144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076379.113576:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616550144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076379.113588:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076379.113592:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076379.113597:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076379.113604:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076379.113623:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076379.113635:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076379/real 1769076379] req@ffff995b8455aa00 x1855007616550272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076379.113658:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616550272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076379.113668:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076379.113672:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076379.113675:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076379.113682:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076379.113699:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076379.113756:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076379/real 1769076379] req@ffff995b84559500 x1855007616550400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076379.113783:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616550400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076379.113795:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076379.113799:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076379.113803:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076379.113811:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616550528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076379.113837:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076379.113851:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076379/real 1769076379] req@ffff995b84558a80 x1855007616550528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076379.113865:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616550528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076379.113883:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076379.113886:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076379.113890:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616550528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076379.113914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616550656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076379.113934:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076379.113946:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076379/real 1769076379] req@ffff995b8455a680 x1855007616550656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076379.113959:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616550656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076379.113968:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076379.113972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076379.113976:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616550656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076379.816591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076379.816666:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076380.136583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076380.136645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076380.136651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076380.136730:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076380.840597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076380.840747:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076381.160578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076381.160664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076381.160732:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076381.160756:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076381.864554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076381.864745:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076382.184562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076382.184562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076382.184563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076382.184619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076382.888570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076382.888678:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076383.208593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076383.208620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076383.208634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076383.208741:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076383.912595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076383.912702:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076384.232594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076384.232663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076384.232684:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076384.232696:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076384.232704:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076384.232730:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076384.232734:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3369 00000100:00080000:3.0:1769076384.232755:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076384.232773:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076384.232776:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076384.232842:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616550784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076384.232891:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076384.232898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076384.232901:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076384.232908:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3369 00000100:00080000:3.0:1769076384.232917:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076384.232939:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076384.232961:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076384.232980:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616550784:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076384.233011:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616550912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076384.233048:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076384.233058:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076384.233064:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076384.233078:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3369 00000100:00080000:3.0:1769076384.233097:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076384.233129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076384.233211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076384/real 1769076384] req@ffff995b8455a680 x1855007616550784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076384.233218:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076384.233243:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076384.233254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616550784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076384.233271:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616551040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076384.233274:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076384.233298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076384.233301:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076384.233311:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1769076384.233317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616550784:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076384.233318:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076384.233370:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076384.233422:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076384.233442:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076384/real 1769076384] req@ffff995b84558a80 x1855007616550912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076384.233524:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616550912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076384.233547:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076384.233560:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076384.233568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616550912:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076384.233582:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3369 00000100:00100000:2.0:1769076384.233590:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551040:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076384.233595:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076384.233608:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076384.233670:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076384.233694:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076384/real 1769076384] req@ffff995b84559500 x1855007616551040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076384.233718:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076384.233724:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616551040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076384.233754:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616551168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076384.233755:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076384.233773:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076384.233788:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551040:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076384.233793:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076384.233801:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076384.233805:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076384.233813:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3369 00000100:00080000:3.0:1769076384.233822:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076384.233825:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551168:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076384.233833:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076384.233844:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076384.233864:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616551296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076384.233880:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076384.233889:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076384.233896:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076384.233899:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076384.233906:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3369 00000100:00080000:3.0:1769076384.233913:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076384.233919:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076384/real 1769076384] req@ffff995b8455aa00 x1855007616551168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076384.233923:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076384.233933:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076384.233965:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616551424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076384.233974:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616551168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076384.234009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076384.234021:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076384.234023:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076384.234027:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076384.234043:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076384.234046:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3369 00000100:00080000:3.0:1769076384.234059:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076384.234062:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551168:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076384.234072:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076384.234085:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076384.234108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076384.234109:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616551552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076384.234169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076384.234190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076384/real 1769076384] req@ffff995b8455bb80 x1855007616551296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076384.234215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616551296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076384.234228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076384.234233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076384.234237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076384.234245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616551424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076384.234266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076384.234278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076384/real 1769076384] req@ffff995b84558000 x1855007616551424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076384.234301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616551424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076384.234318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076384.234325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076384.234331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616551424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076384.234359:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616551552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076384.234400:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076384.234419:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076384/real 1769076384] req@ffff995b8455b100 x1855007616551552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076384.234435:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616551552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076384.234501:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076384.234511:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076384.234517:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616551552:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769076384.936690:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076384.936692:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076385.256708:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076385.256796:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076385.256820:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076385.256832:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076385.960633:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076385.960684:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076386.280562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076386.280672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076386.280699:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076386.280702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076386.984687:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076386.984786:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076387.304630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076387.304630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076387.304716:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076387.304733:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076388.008619:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076388.008680:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076388.328612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076388.328640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076388.328680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076388.328721:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076389.032616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076389.032709:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076389.352587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076389.352607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076389.352662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076389.352687:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076389.352691:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076389.352701:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076389.352707:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076389.352720:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3375 00000100:00080000:2.0:1769076389.352736:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076389.352748:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076389.352793:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616551680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076389.352832:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076389.352837:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076389.352840:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076389.352846:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3375 00000100:00080000:2.0:1769076389.352850:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076389.352857:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076389.352869:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076389.352887:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616551808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076389.352907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616551680:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076389.352910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076389.352941:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076389.352947:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076389.352970:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3375 00000100:00080000:2.0:1769076389.352979:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076389.352987:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076389.353000:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076389.353006:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076389.353042:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616551936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076389.353059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076389.353065:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076389/real 1769076389] req@ffff995b85ac8380 x1855007616551680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076389.353066:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076389.353073:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076389.353088:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3375 00000100:00080000:2.0:1769076389.353100:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076389.353111:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076389.353112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616551680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076389.353118:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769076389.353138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076389.353146:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616552064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076389.353148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076389.353161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616551680:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076389.353183:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076389.353189:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076389.353192:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076389.353197:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3375 00000100:00080000:2.0:1769076389.353202:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076389.353207:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076389.353212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551808:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769076389.353218:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076389.353241:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616552192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076389.353257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076389.353274:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076389.353283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076389/real 1769076389] req@ffff995b85acbb80 x1855007616551808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076389.353287:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076389.353297:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076389.353311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616551808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076389.353332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076389.353332:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3375 00000100:00080000:0.0:1769076389.353348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076389.353349:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076389.353355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551808:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076389.353367:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076389.353371:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551936:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769076389.353381:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076389.353410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076389.353412:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616552320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076389.353432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076389.353433:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076389/real 1769076389] req@ffff995b85acb100 x1855007616551936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076389.353442:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076389.353450:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076389.353497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616551936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076389.353508:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3375 00000100:00080000:2.0:1769076389.353526:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076389.353531:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076389.353559:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076389.353571:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:2.0:1769076389.353572:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076389.353588:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616551936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076389.353601:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616552448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076389.353611:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552064:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076389.353661:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076389.353677:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076389/real 1769076389] req@ffff995b85ac8a80 x1855007616552064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076389.353692:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616552064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076389.353704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076389.353709:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076389.353713:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552064:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076389.353737:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076389.353756:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076389.353769:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076389/real 1769076389] req@ffff995b85aca300 x1855007616552192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076389.353783:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616552192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076389.353794:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076389.353799:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076389.353803:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076389.353809:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616552320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076389.353826:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076389.353838:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076389/real 1769076389] req@ffff995b85acb800 x1855007616552320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076389.353851:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616552320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076389.353862:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076389.353865:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076389.353869:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616552320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076389.353875:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616552448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076389.353892:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076389.353903:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076389/real 1769076389] req@ffff995b85ac9c00 x1855007616552448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076389.353917:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616552448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076389.353927:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076389.353931:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076389.353935:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616552448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076390.056587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076390.056707:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076390.376616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076390.376674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076390.376701:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076390.376740:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076391.080566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076391.080706:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076391.400636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076391.400638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076391.400653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076391.400712:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076392.104596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076392.104734:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076392.424626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076392.424668:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076392.424670:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076392.424701:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076393.128567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076393.128783:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076393.448628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076393.448656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076393.448724:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076393.448745:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076394.152575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076394.152712:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076394.472614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076394.472614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076394.472639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076394.472646:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076394.472661:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076394.472670:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076394.472673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076394.472693:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3380 00000100:00080000:1.0:1769076394.472718:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076394.472732:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076394.472799:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616552576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076394.472844:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076394.472850:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076394.472854:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076394.472860:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3380 00000100:00080000:1.0:1769076394.472864:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076394.472871:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076394.472885:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076394.472905:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616552704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076394.472916:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616552576:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076394.472946:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076394.472952:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076394.472955:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076394.472962:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3380 00000100:00080000:1.0:1769076394.472966:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076394.472970:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076394.472978:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076394.472997:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616552832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076394.473008:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076394.473013:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076394.473017:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076394.473022:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3380 00000100:00080000:1.0:1769076394.473040:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076394.473065:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769076394.473066:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769076394.473087:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076394.473110:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616552960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076394.473125:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076394.473131:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076394.473135:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076394.473144:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3380 00000100:00100000:0.0:1769076394.473146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076394/real 1769076394] req@ffff995b868c4a80 x1855007616552576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076394.473158:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076394.473176:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076394.473197:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076394.473229:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616553088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076394.473232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616552576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076394.473319:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076394.473325:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076394.473327:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076394.473339:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076394.473347:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076394.473358:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616552576:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076394.473366:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3380 00000100:00080000:1.0:1769076394.473390:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076394.473399:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076394.473406:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076394.473426:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616553216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076394.473430:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552704:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076394.473567:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769076394.473574:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076394.473586:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076394.473593:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076394.473603:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3380 00000100:00080000:1.0:1769076394.473609:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076394.473613:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076394/real 1769076394] req@ffff995b868c7800 x1855007616552704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076394.473622:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076394.473636:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076394.473665:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616553344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076394.473682:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616552704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076394.473709:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076394.473720:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076394.473729:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076394.473747:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076394.473783:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076394.473799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076394/real 1769076394] req@ffff995b868c5500 x1855007616552832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076394.473815:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616552832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076394.473828:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076394.473832:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076394.473837:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076394.473845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552960:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076394.473866:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076394.473878:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076394/real 1769076394] req@ffff995b868c5c00 x1855007616552960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076394.473892:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616552960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076394.473904:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076394.473907:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076394.473911:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616552960:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076394.473919:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076394.473937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076394.473948:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076394/real 1769076394] req@ffff995b868c6d80 x1855007616553088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076394.473962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616553088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076394.473974:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076394.473977:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076394.473981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076394.474008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616553216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076394.474027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076394.474039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076394/real 1769076394] req@ffff995b868c6300 x1855007616553216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076394.474054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616553216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076394.474064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076394.474068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076394.474072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616553216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076394.474079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616553344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076394.474097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076394.474108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076394/real 1769076394] req@ffff995b868c7100 x1855007616553344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076394.474122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616553344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076394.474140:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076394.474144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076394.474148:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616553344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076395.176578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076395.176671:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076395.496580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076395.496641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076395.496667:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076395.496667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076396.200523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076396.200630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076396.520594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076396.520652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076396.520686:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076396.520691:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076397.224525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076397.224653:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076397.544573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076397.544686:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076397.544691:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076397.544731:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076398.248562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076398.248717:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076398.568628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076398.568641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076398.568686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076398.568711:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076399.272587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076399.272687:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076399.592598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076399.592615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076399.592617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076399.592628:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076399.592640:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076399.592647:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076399.592665:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3385 00000100:00100000:2.0:1769076399.592676:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076399.592682:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076399.592695:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076399.592773:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616553472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076399.592811:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076399.592816:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076399.592819:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076399.592824:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3385 00000100:00080000:1.0:1769076399.592830:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076399.592836:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076399.592849:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076399.592870:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616553600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076399.592891:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076399.592902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616553472:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076399.592902:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076399.592910:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076399.592922:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3385 00000100:00080000:1.0:1769076399.592929:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076399.592938:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076399.592964:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076399.592994:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616553728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076399.593017:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076399.593025:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076399.593029:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076399.593039:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3385 00000100:00080000:1.0:1769076399.593047:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076399.593055:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076399.593067:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076399.593073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076399.593112:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616553856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076399.593128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553728:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076399.593138:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076399.593148:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076399.593152:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076399.593163:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3385 00000100:00100000:0.0:1769076399.593171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553856:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769076399.593175:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076399.593188:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076399.593213:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076399.593236:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616553984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076399.593268:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076399.593276:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076399.593279:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076399.593289:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3385 00000100:00080000:1.0:1769076399.593298:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076399.593306:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076399.593318:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076399.593322:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076399.593387:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616554112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076399.593407:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076399.593412:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076399.593415:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076399.593427:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3385 00000100:00100000:0.0:1769076399.593429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616554112:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076399.593434:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076399.593446:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076399.593487:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076399.593514:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616554240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076399.593568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616554240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076400.296579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076400.296698:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076400.616658:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076400.616697:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076400.616743:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076400.616776:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076401.320566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076401.320643:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076401.640583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076401.640679:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076401.640708:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076401.640716:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076402.344536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076402.344628:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076402.664611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076402.664617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076402.664645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076402.664768:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:1.0:1769076402.664869:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076402.664916:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076402.664936:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076402.664978:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076402.664991:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076402.665004:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076402.665016:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076402.665027:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076402.665036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076399/real 1769076402] req@ffff995b868c6300 x1855007616553984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1769076402.665038:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076402.665068:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616553984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076402.665087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076402.665094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076402.665101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076402.665111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076399/real 1769076402] req@ffff995b868c7100 x1855007616553856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076402.665131:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616553856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076402.665145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076402.665150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076402.665157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076402.665166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076399/real 1769076402] req@ffff995b868c6a00 x1855007616553600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076402.665200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616553600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076402.665212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076402.665218:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076402.665223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076402.665233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076399/real 1769076402] req@ffff995b868c4700 x1855007616553728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076402.665250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616553728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076402.665264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076402.665269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076402.665273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616553728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076402.665282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076399/real 1769076402] req@ffff995b868c7480 x1855007616553472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076402.665299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616553472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076402.665309:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076402.665364:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076402.665371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616553472:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076402.665407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076399/real 1769076402] req@ffff995b868c5c00 x1855007616554240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076402.665425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616554240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076402.665438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076402.665443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076402.665447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616554240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076402.665487:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076399/real 1769076402] req@ffff995b868c6d80 x1855007616554112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076402.665505:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616554112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076402.665516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076402.665521:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076402.665526:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616554112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076403.368549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076403.368622:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076403.688522:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076403.688560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076403.688574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076403.688598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076404.392550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076404.392682:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076404.712528:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076404.712540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076404.712547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076404.712580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076404.712594:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076404.712601:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076404.712605:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076404.712615:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3390 00000100:00080000:0.0:1769076404.712626:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076404.712632:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076404.712662:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616554368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076404.712690:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076404.712694:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076404.712696:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076404.712701:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3390 00000100:00080000:0.0:1769076404.712706:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076404.712711:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076404.712720:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616554368:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076404.712722:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076404.712741:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616554496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076404.712760:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076404.712763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076404.712765:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076404.712769:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3390 00000100:00080000:0.0:1769076404.712777:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076404.712783:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076404.712789:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769076404.712793:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076404.712809:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616554624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076404.712820:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076404.712823:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076404.712826:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076404.712832:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3390 00000100:00100000:2.0:1769076404.712833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076404/real 1769076404] req@ffff995b84643800 x1855007616554368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076404.712837:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076404.712846:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076404.712856:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076404.712864:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616554368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076404.712882:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616554752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076404.712892:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076404.712898:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076404.712903:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076404.712905:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076404.712910:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076404.712911:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616554368:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076404.712918:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3390 00000100:00080000:0.0:1769076404.712924:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076404.712929:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076404.712934:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076404.712943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616554496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076404.712949:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616554880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076404.712963:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076404.712967:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076404.712970:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076404.712977:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3390 00000800:00000100:2.0:1769076404.712977:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076404.712990:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076404.713001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076404/real 1769076404] req@ffff995b84642a00 x1855007616554496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076404.713002:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076404.713015:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076404.713023:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616554496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076404.713031:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616555008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076404.713049:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076404.713057:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076404.713060:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076404.713067:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076404.713068:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3390 00000100:00080000:0.0:1769076404.713077:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076404.713079:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076404.713084:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076404.713087:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616554496:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769076404.713094:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076404.713096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616554624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076404.713117:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616555136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076404.713127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076404.713144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076404/real 1769076404] req@ffff995b84640e00 x1855007616554624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076404.713160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616554624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076404.713173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076404.713177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076404.713180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616554624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076404.713186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616554752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076404.713207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076404.713226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076404/real 1769076404] req@ffff995b84640000 x1855007616554752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076404.713239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616554752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076404.713247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076404.713250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076404.713253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616554752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076404.713270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616554880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076404.713325:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076404.713337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076404/real 1769076404] req@ffff995b84641c00 x1855007616554880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076404.713351:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616554880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076404.713361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076404.713364:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076404.713367:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616554880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076404.713374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616555008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076404.713394:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076404.713407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076404/real 1769076404] req@ffff995b84642680 x1855007616555008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076404.713419:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616555008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076404.713428:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076404.713432:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076404.713435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616555008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076404.713442:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616555136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076404.713483:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076404.713492:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076404/real 1769076404] req@ffff995b84640700 x1855007616555136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076404.713501:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616555136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076404.713507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076404.713509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076404.713511:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616555136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076405.416593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076405.416671:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076405.736606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076405.736616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076405.736698:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076405.736740:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076406.440581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076406.440762:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076406.760647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076406.760682:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076406.760704:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076406.760754:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076407.464579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076407.464679:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076407.784605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076407.784659:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076407.784673:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076407.784711:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076408.488558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076408.488739:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076408.808612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076408.808612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076408.808632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076408.808671:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076409.512588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076409.512668:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076409.832580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076409.832602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076409.832623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076409.832630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076409.832685:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076409.832695:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076409.832702:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076409.832715:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3395 00000100:00080000:0.0:1769076409.832730:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076409.832740:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076409.832784:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616555264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076409.832822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076409.832827:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076409.832829:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076409.832835:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3395 00000100:00080000:0.0:1769076409.832839:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076409.832844:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076409.832856:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076409.832876:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616555392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076409.832899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076409.832900:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616555264:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076409.832920:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076409.832926:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076409.832936:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3395 00000100:00080000:0.0:1769076409.832944:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076409.832952:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076409.832964:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076409.832988:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616555520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076409.833007:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769076409.833017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076409.833019:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076409.833027:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076409.833035:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3395 00000100:00080000:0.0:1769076409.833042:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076409.833050:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076409.833058:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076409.833078:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616555648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076409.833086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076409/real 1769076409] req@ffff995b84640a80 x1855007616555264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076409.833105:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076409.833118:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076409.833123:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076409.833133:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3395 00000100:00080000:0.0:1769076409.833138:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076409.833145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616555264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076409.833148:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076409.833160:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769076409.833173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769076409.833182:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616555776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076409.833186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076409.833229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616555264:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076409.833240:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076409.833250:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076409.833255:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076409.833266:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3395 00000100:00080000:0.0:1769076409.833275:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076409.833276:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616555392:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076409.833293:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076409.833307:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076409.833330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076409.833343:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616555904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076409.833358:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076409/real 1769076409] req@ffff995b84641f80 x1855007616555392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076409.833368:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076409.833383:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076409.833390:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076409.833404:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3395 00000100:00080000:0.0:1769076409.833413:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076409.833422:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076409.833424:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616555392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076409.833430:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076409.833488:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616556032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076409.833510:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076409.833521:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076409.833529:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616555392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076409.833542:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616555520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076409.833579:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076409.833597:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076409/real 1769076409] req@ffff995b84640380 x1855007616555520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076409.833613:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616555520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076409.833626:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076409.833631:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076409.833635:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616555520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076409.833643:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616555648:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076409.833662:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076409.833675:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076409/real 1769076409] req@ffff995b84640700 x1855007616555648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076409.833689:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616555648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076409.833701:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076409.833704:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076409.833708:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616555648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076409.833715:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616555776:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076409.833736:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076409.833755:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076409/real 1769076409] req@ffff995b84642680 x1855007616555776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076409.833778:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616555776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076409.833796:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076409.833803:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076409.833810:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616555776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076409.833848:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616555904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076409.833885:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076409.833901:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076409/real 1769076409] req@ffff995b84641c00 x1855007616555904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076409.833916:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616555904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076409.833928:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076409.833932:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076409.833936:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616555904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076409.833944:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616556032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076409.833963:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076409.833974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076409/real 1769076409] req@ffff995b84640000 x1855007616556032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076409.833988:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616556032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076409.833998:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076409.834001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076409.834005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616556032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076410.536574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076410.536702:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076410.856591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076410.856620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076410.856633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076410.856646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076411.560571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076411.560753:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076411.880627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076411.880791:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076411.880821:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076411.881115:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076412.584570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076412.584639:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076412.904531:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076412.904534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076412.904615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076412.904683:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076413.608571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076413.608643:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076413.928541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076413.928620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076413.928629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076413.928640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076414.632573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076414.632688:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076414.952582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076414.952606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076414.952624:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076414.952628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076414.952637:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076414.952646:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076414.952657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076414.952660:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3400 00000100:00080000:1.0:1769076414.952680:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076414.952691:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076414.952737:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616556160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076414.952773:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076414.952778:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076414.952781:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076414.952788:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3400 00000100:00080000:1.0:1769076414.952795:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076414.952804:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076414.952820:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076414.952839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616556160:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076414.952851:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616556288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076414.952882:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076414.952889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076414.952894:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076414.952903:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3400 00000100:00080000:1.0:1769076414.952914:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076414.952923:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076414.952932:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076414.952950:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076414.952958:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616556416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076414.952976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076414.952981:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076414.952983:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076414.952989:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3400 00000100:00080000:1.0:1769076414.952995:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076414.953001:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076414.953009:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076414.953018:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076414/real 1769076414] req@ffff995b868c5500 x1855007616556160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076414.953032:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616556544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076414.953069:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076414.953074:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076414.953076:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076414.953088:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3400 00000100:00100000:2.0:1769076414.953090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616556160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076414.953096:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076414.953135:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076414.953146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769076414.953151:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769076414.953159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076414.953168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616556160:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076414.953201:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616556672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076414.953206:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616556288:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076414.953234:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076414.953240:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076414.953243:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076414.953250:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3400 00000800:00000100:2.0:1769076414.953260:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076414.953263:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076414.953274:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076414.953284:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076414.953295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076414/real 1769076414] req@ffff995b868c7800 x1855007616556288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076414.953306:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616556800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076414.953370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616556288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076414.953377:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076414.953398:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076414.953404:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076414.953406:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076414.953424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076414.953426:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3400 00000100:00080000:1.0:1769076414.953440:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076414.953443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616556288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076414.953508:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616556416:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076414.953518:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076414.953535:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076414.953567:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076414.953571:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616556928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076414.953593:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076414/real 1769076414] req@ffff995b868c4a80 x1855007616556416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076414.953614:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616556416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076414.953631:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076414.953636:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076414.953643:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616556416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076414.953657:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616556544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076414.953696:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076414.953720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076414/real 1769076414] req@ffff995b868c6d80 x1855007616556544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076414.953742:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616556544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076414.953754:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076414.953758:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076414.953763:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616556544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076414.953786:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616556672:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076414.953813:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076414.953827:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076414/real 1769076414] req@ffff995b868c5c00 x1855007616556672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076414.953841:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616556672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076414.953853:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076414.953857:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076414.953861:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616556672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076414.953867:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616556800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076414.953885:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076414.953897:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076414/real 1769076414] req@ffff995b868c7480 x1855007616556800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076414.953910:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616556800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076414.953921:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076414.953924:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076414.953927:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616556800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076414.953934:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616556928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076414.953951:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076414.953962:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076414/real 1769076414] req@ffff995b868c4700 x1855007616556928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076414.953975:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616556928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076414.953985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076414.953988:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076414.953995:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616556928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076415.656550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076415.656673:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076415.976616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076415.976634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076415.976667:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076415.976725:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076416.680561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076416.680714:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076417.000553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076417.000614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076417.000633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076417.000643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076417.704573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076417.704662:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076418.024606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076418.024617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076418.024693:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076418.024699:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076418.728588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076418.728681:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076419.048586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076419.048603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076419.048628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076419.048641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076419.752561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076419.752674:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076420.072584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076420.072621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076420.072622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076420.072657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076420.072706:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076420.072714:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076420.072720:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076420.072732:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3405 00000100:00080000:2.0:1769076420.072745:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076420.072756:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076420.072806:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616557056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076420.072852:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076420.072861:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076420.072866:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076420.072875:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3405 00000100:00080000:2.0:1769076420.072884:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076420.072893:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076420.072913:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076420.072921:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616557056:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076420.072949:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616557184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076420.072973:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076420.072978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076420.072982:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076420.072990:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3405 00000100:00080000:2.0:1769076420.072999:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076420.073006:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769076420.073016:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769076420.073017:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076420.073073:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616557312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076420.073090:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076420.073096:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076420.073099:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076420.073105:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3405 00000100:00080000:2.0:1769076420.073114:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076420.073121:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076420.073129:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076420.073145:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616557440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076420.073157:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076420.073161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076420.073164:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076420.073170:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3405 00000100:00080000:2.0:1769076420.073175:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076420.073180:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076420.073185:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076420.073198:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616557568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076420.073209:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076420/real 1769076420] req@ffff995b85acaa00 x1855007616557056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076420.073215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076420.073230:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076420.073237:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076420.073246:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3405 00000100:00080000:2.0:1769076420.073252:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076420.073258:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076420.073268:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076420.073275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616557056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076420.073284:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616557696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076420.073317:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076420.073319:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076420.073328:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076420.073331:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076420.073341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616557056:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076420.073350:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076420.073358:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3405 00000100:00080000:2.0:1769076420.073364:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076420.073368:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076420.073374:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076420.073383:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616557184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076420.073397:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616557824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076420.073431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076420.073494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076420/real 1769076420] req@ffff995b85ac9880 x1855007616557184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076420.073519:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616557184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076420.073537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076420.073544:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076420.073551:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616557184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076420.073561:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616557312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076420.073592:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076420.073611:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076420/real 1769076420] req@ffff995b85ac9c00 x1855007616557312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076420.073637:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616557312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076420.073661:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076420.073667:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076420.073672:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616557312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076420.073681:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616557440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076420.073700:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076420.073713:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076420/real 1769076420] req@ffff995b85acb800 x1855007616557440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076420.073728:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616557440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076420.073739:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076420.073742:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076420.073746:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616557440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076420.073753:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616557568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076420.073770:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076420.073782:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076420/real 1769076420] req@ffff995b85aca300 x1855007616557568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076420.073796:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616557568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076420.073807:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076420.073810:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076420.073814:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616557568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076420.073820:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616557696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076420.073838:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076420.073849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076420/real 1769076420] req@ffff995b85ac8a80 x1855007616557696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076420.073863:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616557696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076420.073873:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076420.073877:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076420.073880:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616557696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076420.073903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616557824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076420.073922:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076420.073933:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076420/real 1769076420] req@ffff995b85acb100 x1855007616557824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076420.073946:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616557824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076420.073958:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076420.073961:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076420.073965:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616557824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076420.776570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076420.776675:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076421.096617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076421.096632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076421.096641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076421.096687:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076421.800601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076421.800675:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076422.120613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076422.120619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076422.120634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076422.120644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076422.824585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076422.824745:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076423.144560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076423.144620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076423.144634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076423.144658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076423.848589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076423.848694:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076424.168564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076424.168609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076424.168672:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076424.168703:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076424.872580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076424.872671:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076425.192585:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076425.192600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076425.192603:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076425.192613:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076425.192626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076425.192634:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3410 00000100:00100000:0.0:1769076425.192653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076425.192656:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076425.192675:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076425.192691:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076425.192734:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616557952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076425.192784:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076425.192793:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076425.192797:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076425.192808:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3410 00000100:00080000:2.0:1769076425.192817:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076425.192827:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076425.192836:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616557952:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769076425.192851:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076425.192879:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616558080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076425.192896:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076425.192900:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076425.192903:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076425.192908:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3410 00000100:00080000:2.0:1769076425.192916:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076425.192921:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076425.192927:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076425.192967:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076425.192988:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616558208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076425.193004:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076425.193011:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076425.193016:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076425.193025:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076425/real 1769076425] req@ffff995b85acbb80 x1855007616557952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076425.193032:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3410 00000100:00080000:2.0:1769076425.193047:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076425.193061:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076425.193065:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616557952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076425.193081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076425.193093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:2.0:1769076425.193099:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076425.193100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616557952:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076425.193136:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616558336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076425.193145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558080:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076425.193170:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076425.193177:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076425.193182:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076425.193189:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3410 00000800:00000100:0.0:1769076425.193199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076425.193205:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076425.193216:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076425.193225:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076425/real 1769076425] req@ffff995b85ac8380 x1855007616558080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076425.193225:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076425.193254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616558080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076425.193264:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616558464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076425.193328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076425.193333:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076425.193339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076425.193351:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769076425.193352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558080:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076425.193359:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076425.193369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558208:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076425.193377:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3410 00000100:00080000:2.0:1769076425.193391:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076425.193403:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076425.193405:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076425.193423:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076425.193426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076425/real 1769076425] req@ffff995b85acb100 x1855007616558208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076425.193447:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616558208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076425.193489:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076425.193494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076425.193501:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076425.193513:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616558592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076425.193526:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558336:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769076425.193540:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076425.193547:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076425.193550:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076425.193558:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3410 00000800:00000100:0.0:1769076425.193565:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076425.193575:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076425.193587:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076425/real 1769076425] req@ffff995b85ac8a80 x1855007616558336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076425.193592:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076425.193614:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076425.193619:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616558336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076425.193639:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076425.193644:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076425.193654:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076425.193656:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616558720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076425.193665:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558464:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076425.193696:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076425.193713:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076425/real 1769076425] req@ffff995b85aca300 x1855007616558464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076425.193730:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616558464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076425.193741:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076425.193744:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076425.193748:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076425.193762:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616558592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076425.193781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076425.193792:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076425/real 1769076425] req@ffff995b85acb800 x1855007616558592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076425.193806:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616558592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076425.193816:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076425.193819:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076425.193823:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616558592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076425.193829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616558720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076425.193845:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076425.193856:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076425/real 1769076425] req@ffff995b85ac9c00 x1855007616558720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076425.193870:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616558720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076425.193880:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076425.193884:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076425.193887:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616558720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076425.896574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076425.896686:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076426.216585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076426.216601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076426.216658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076426.216684:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076426.920606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076426.920661:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076427.240604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076427.240621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076427.240646:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076427.240741:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076427.944594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076427.944740:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076428.264588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076428.264622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076428.264629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076428.264805:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076428.968610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076428.968724:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076429.288591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076429.288602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076429.288615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076429.288649:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076429.992590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076429.992667:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076430.312619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076430.312625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076430.312639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076430.312646:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076430.312649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076430.312662:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3416 00000100:00100000:2.0:1769076430.312667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076430.312678:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076430.312688:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076430.312694:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076430.312745:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616558848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076430.312785:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076430.312790:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076430.312793:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076430.312798:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3416 00000100:00080000:0.0:1769076430.312803:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076430.312808:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076430.312821:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076430.312838:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616558976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076430.312880:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076430.312882:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616558848:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076430.312887:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076430.312895:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076430.312904:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3416 00000100:00080000:0.0:1769076430.312908:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076430.312914:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076430.312923:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076430.312942:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616559104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076430.312955:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076430.312960:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076430.312962:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076430.312968:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3416 00000100:00080000:0.0:1769076430.312972:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076430.312976:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076430.312984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769076430.312985:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076430.313005:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616559232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076430.313017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076430.313023:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076430.313025:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076430.313032:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3416 00000100:00080000:0.0:1769076430.313038:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076430.313043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076430/real 1769076430] req@ffff995b84640e00 x1855007616558848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076430.313049:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076430.313062:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076430.313084:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616559360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076430.313091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616558848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076430.313116:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076430.313132:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076430.313135:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076430.313138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076430.313144:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3416 00000100:00080000:0.0:1769076430.313154:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076430.313160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076430.313165:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076430.313175:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076430.313180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616558848:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076430.313193:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616559488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076430.313207:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076430.313214:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076430.313218:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076430.313227:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3416 00000100:00100000:2.0:1769076430.313233:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558976:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076430.313239:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076430.313252:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076430.313265:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076430.313282:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076430.313298:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616559616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076430.313309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076430/real 1769076430] req@ffff995b84642a00 x1855007616558976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076430.313369:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616558976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076430.313381:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076430.313387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076430.313393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616558976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076430.313402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616559104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076430.313424:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076430.313437:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076430/real 1769076430] req@ffff995b84643800 x1855007616559104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076430.313451:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616559104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076430.313497:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076430.313502:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076430.313506:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616559104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076430.313515:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616559232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076430.313544:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076430.313566:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076430/real 1769076430] req@ffff995b84641180 x1855007616559232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076430.313590:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616559232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076430.313622:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076430.313629:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076430.313636:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616559232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076430.313649:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616559360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076430.313684:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076430.313708:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076430/real 1769076430] req@ffff995b84641500 x1855007616559360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076430.313731:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616559360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076430.313750:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076430.313757:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076430.313764:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616559360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076430.313775:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616559488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076430.313815:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076430.313838:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076430/real 1769076430] req@ffff995b84643b80 x1855007616559488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076430.313885:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616559488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076430.313897:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076430.313900:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076430.313905:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616559488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076430.313935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616559616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076430.313964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076430.313978:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076430/real 1769076430] req@ffff995b84641880 x1855007616559616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076430.313993:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616559616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076430.314003:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076430.314007:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076430.314012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616559616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076431.016582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076431.016775:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076431.336670:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076431.336736:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076431.336817:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076431.336901:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076432.040582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076432.040701:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076432.360576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076432.360648:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076432.360731:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076432.360772:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076433.064594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076433.064670:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076433.384589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076433.384595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076433.384612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076433.384638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076434.088593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076434.088655:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076434.408575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076434.408627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076434.408642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076434.408660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076435.112604:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076435.112733:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076435.432598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076435.432685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076435.432686:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076435.432700:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769076435.432704:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076435.432709:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076435.432710:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076435.432734:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3421 00000100:00080000:1.0:1769076435.432751:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076435.432786:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076435.432847:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616559744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076435.432884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076435.432889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076435.432892:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076435.432897:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3421 00000100:00080000:1.0:1769076435.432902:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076435.432908:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076435.432918:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076435.432936:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616559872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076435.432950:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076435.432955:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076435.432958:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076435.432968:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3421 00000100:00100000:2.0:1769076435.432970:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616559744:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076435.432972:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076435.432984:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076435.432993:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076435.433007:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616560000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076435.433028:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076435.433033:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076435.433036:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076435.433040:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3421 00000100:00080000:1.0:1769076435.433044:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076435.433048:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076435.433054:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076435.433069:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616560128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076435.433078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076435.433095:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076435.433100:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076435.433104:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076435.433111:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3421 00000100:00080000:1.0:1769076435.433115:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076435.433119:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076435.433125:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076435.433141:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616560256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076435.433148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076435/real 1769076435] req@ffff995b868c6a00 x1855007616559744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076435.433178:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076435.433187:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076435.433190:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076435.433199:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3421 00000100:00100000:2.0:1769076435.433205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616559744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076435.433207:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076435.433223:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076435.433236:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076435.433241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076435.433255:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616560384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076435.433256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076435.433271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616559744:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076435.433288:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076435.433295:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076435.433297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616559872:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076435.433299:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076435.433314:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3421 00000100:00080000:1.0:1769076435.433318:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076435.433323:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076435.433329:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076435.433344:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616560512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076435.433362:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076435.433400:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076435/real 1769076435] req@ffff995b868c7100 x1855007616559872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076435.433430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616559872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076435.433448:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076435.433503:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076435.433512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616559872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076435.433555:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076435.433594:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076435.433617:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076435/real 1769076435] req@ffff995b868c6300 x1855007616560000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076435.433641:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616560000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076435.433674:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076435.433681:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076435.433688:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076435.433700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560128:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076435.433740:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076435.433791:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076435/real 1769076435] req@ffff995b868c4700 x1855007616560128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076435.433817:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616560128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076435.433836:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076435.433842:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076435.433848:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076435.433861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076435.433905:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076435.433929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076435/real 1769076435] req@ffff995b868c7480 x1855007616560256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076435.433946:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616560256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076435.433958:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076435.433961:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076435.433966:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076435.433975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616560384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076435.434001:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076435.434014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076435/real 1769076435] req@ffff995b868c5c00 x1855007616560384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076435.434028:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616560384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076435.434038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076435.434042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076435.434045:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616560384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076435.434053:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616560512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076435.434072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076435.434083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076435/real 1769076435] req@ffff995b868c6d80 x1855007616560512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076435.434097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616560512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076435.434107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076435.434111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076435.434114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616560512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076436.136568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076436.136695:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076436.456573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076436.456623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076436.456623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076436.456648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076437.160615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076437.160678:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076437.480664:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076437.480670:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076437.480993:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076437.480994:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076438.184567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076438.184655:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076438.504612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076438.504624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076438.504632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076438.504658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076439.208554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076439.208797:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076439.528521:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076439.528547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076439.528577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076439.528591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076440.232573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076440.232804:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076440.552597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076440.552613:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076440.552618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076440.552631:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769076440.552636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076440.552639:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076440.552659:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3426 00000100:00100000:1.0:1769076440.552667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076440.552675:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076440.552724:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076440.552781:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616560640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076440.552803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076440.552806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076440.552808:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076440.552811:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3426 00000100:00080000:3.0:1769076440.552814:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076440.552818:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076440.552826:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076440.552838:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616560640:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076440.552839:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616560768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076440.552859:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076440.552862:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076440.552864:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076440.552867:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3426 00000100:00080000:3.0:1769076440.552870:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076440.552872:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076440.552876:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076440.552885:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616560896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076440.552893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076440.552896:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076440.552897:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076440.552900:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3426 00000100:00080000:3.0:1769076440.552902:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076440.552907:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076440.552908:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076440.552912:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076440.552923:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616561024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076440.552930:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076440.552933:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076440.552934:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076440.552953:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3426 00000100:00080000:3.0:1769076440.552961:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076440.552966:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076440.552974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076440/real 1769076440] req@ffff995b84558700 x1855007616560640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076440.552977:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076440.552999:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616561152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076440.553011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616560640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076440.553015:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076440.553035:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076440.553043:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076440.553048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076440.553050:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076440.553059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616560640:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076440.553061:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3426 00000100:00080000:3.0:1769076440.553074:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076440.553079:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076440.553085:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076440.553099:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616561280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076440.553103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560768:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076440.553130:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076440.553135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076440.553138:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076440.553144:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3426 00000100:00080000:3.0:1769076440.553150:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076440.553157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076440.553163:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076440.553172:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076440.553186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076440/real 1769076440] req@ffff995b8455b480 x1855007616560768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076440.553203:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616561408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076440.553209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616560768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076440.553264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076440.553272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076440.553278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076440.553288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076440.553307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076440.553315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076440/real 1769076440] req@ffff995b84559880 x1855007616560896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076440.553324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616560896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076440.553332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076440.553334:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076440.553337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616560896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076440.553341:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561024:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076440.553353:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076440.553360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076440/real 1769076440] req@ffff995b8455ad80 x1855007616561024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076440.553368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616561024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076440.553375:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076440.553377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076440.553380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076440.553383:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561152:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076440.553393:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076440.553400:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076440/real 1769076440] req@ffff995b8455a300 x1855007616561152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076440.553407:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616561152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076440.553414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076440.553416:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076440.553418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076440.553430:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616561280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076440.553441:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076440.553448:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076440/real 1769076440] req@ffff995b84559c00 x1855007616561280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076440.553456:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616561280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076440.553462:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076440.553464:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076440.553466:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616561280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076440.553470:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616561408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076440.553481:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076440.553541:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076440/real 1769076440] req@ffff995b84558700 x1855007616561408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076440.553556:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616561408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076440.553570:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076440.553575:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076440.553582:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616561408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076441.256596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076441.256746:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076441.576552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076441.576557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076441.576615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076441.576674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076442.280575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076442.280693:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076442.600594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076442.600605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076442.600640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076442.600640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076443.304585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076443.304719:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076443.624577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076443.624578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076443.624616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076443.624678:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076444.328592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076444.328798:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076444.648544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076444.648559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076444.648560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076444.648589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076445.352594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076445.352766:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076445.672558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076445.672561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076445.672609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076445.672637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076445.672647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076445.672652:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076445.672664:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3431 00000100:00100000:1.0:1769076445.672667:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076445.672678:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076445.672687:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076445.672725:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616561536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076445.672748:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076445.672752:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076445.672754:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076445.672761:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3431 00000100:00080000:0.0:1769076445.672764:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076445.672767:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076445.672774:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076445.672790:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616561664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076445.672790:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616561536:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076445.672812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076445.672815:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076445.672816:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076445.672820:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3431 00000100:00080000:0.0:1769076445.672824:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076445.672827:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076445.672830:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076445.672839:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616561792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076445.672846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076445.672848:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076445.672849:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076445.672851:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3431 00000100:00080000:0.0:1769076445.672854:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076445.672857:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076445.672861:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076445.672869:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616561920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076445.672876:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076445.672879:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076445.672880:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076445.672891:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3431 00000100:00080000:0.0:1769076445.672893:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076445.672895:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076445.672898:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076445.672899:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076445.672911:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616562048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076445.672920:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076445.672923:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076445.672925:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076445.672928:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3431 00000100:00080000:0.0:1769076445.672930:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076445.672933:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076445.672937:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076445.672949:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616562176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076445.672957:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076445/real 1769076445] req@ffff995b84642d80 x1855007616561536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076445.672960:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076445.672975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076445.672983:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076445.672994:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3431 00000100:00100000:2.0:1769076445.672994:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616561536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076445.673010:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076445.673014:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076445.673021:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076445.673021:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076445.673031:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616561536:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076445.673038:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076445.673055:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616562304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076445.673067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076445.673088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076445.673099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076445/real 1769076445] req@ffff995b84643100 x1855007616561664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076445.673107:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616561664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076445.673113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076445.673116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076445.673118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076445.673122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076445.673133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076445.673139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076445/real 1769076445] req@ffff995b84643480 x1855007616561792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076445.673147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616561792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076445.673155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076445.673157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076445.673158:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076445.673163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076445.673174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076445.673181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076445/real 1769076445] req@ffff995b84641880 x1855007616561920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076445.673188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616561920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076445.673195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076445.673196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076445.673198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616561920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076445.673202:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076445.673213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076445.673219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076445/real 1769076445] req@ffff995b84643b80 x1855007616562048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076445.673226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616562048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076445.673233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076445.673235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076445.673236:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076445.673240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616562176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076445.673250:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076445.673256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076445/real 1769076445] req@ffff995b84641500 x1855007616562176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076445.673262:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616562176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076445.673269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076445.673271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076445.673272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616562176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076445.673286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616562304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076445.673296:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076445.673302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076445/real 1769076445] req@ffff995b84641180 x1855007616562304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076445.673309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616562304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076445.673315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076445.673316:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076445.673318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616562304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076446.376602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076446.376786:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076446.696522:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076446.696543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076446.696593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076446.696613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076447.400657:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076447.400770:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076447.720552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076447.720581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076447.720717:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076447.720721:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076448.424596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076448.424729:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076448.744628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076448.744630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076448.744631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076448.744665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076449.448608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076449.448713:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076449.768562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076449.768588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076449.768593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076449.768622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076450.472607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076450.472719:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076450.792620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076450.792620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076450.792627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076450.792655:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076450.792662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076450.792668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076450.792672:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076450.792680:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3436 00000100:00080000:1.0:1769076450.792690:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076450.792696:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076450.792729:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616562432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076450.792749:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076450.792751:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076450.792753:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076450.792756:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3436 00000100:00080000:1.0:1769076450.792758:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076450.792763:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076450.792770:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076450.792780:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616562432:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076450.792781:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616562560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076450.792793:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076450.792795:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076450.792796:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076450.792798:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3436 00000100:00080000:1.0:1769076450.792801:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076450.792805:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076450.792810:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076450.792823:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616562688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076450.792845:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076450.792848:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076450.792850:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076450.792857:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3436 00000800:00000100:2.0:1769076450.792857:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076450.792864:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076450.792870:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076450.792894:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076450.792919:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616562816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076450.792933:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076450/real 1769076450] req@ffff995b868c4a80 x1855007616562432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076450.792937:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076450.792952:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076450.792957:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076450.792971:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3436 00000100:00080000:1.0:1769076450.792979:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076450.792988:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076450.792990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616562432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076450.792995:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769076450.793033:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076450.793035:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616562944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076450.793042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076450.793054:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616562432:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076450.793057:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076450.793062:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076450.793065:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076450.793069:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3436 00000100:00080000:1.0:1769076450.793072:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076450.793077:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076450.793083:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076450.793084:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076450.793107:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616563072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076450.793120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076450.793130:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076450.793136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076450.793138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076450/real 1769076450] req@ffff995b868c7800 x1855007616562560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076450.793140:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076450.793154:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3436 00000100:00100000:2.0:1769076450.793161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616562560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076450.793164:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076450.793177:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076450.793177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076450.793189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769076450.793191:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076450.793193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076450.793221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076450.793229:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616563200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076450.793238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076450.793250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076450/real 1769076450] req@ffff995b868c5500 x1855007616562688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076450.793259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616562688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076450.793270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076450.793272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076450.793274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076450.793278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562816:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076450.793311:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076450.793323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076450/real 1769076450] req@ffff995b868c6d80 x1855007616562816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076450.793338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616562816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076450.793349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076450.793352:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076450.793356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076450.793363:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562944:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076450.793381:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076450.793398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076450/real 1769076450] req@ffff995b868c5c00 x1855007616562944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076450.793417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616562944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076450.793434:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076450.793440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076450.793447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616562944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076450.793489:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616563072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076450.793572:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076450.793582:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076450/real 1769076450] req@ffff995b868c7480 x1855007616563072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076450.793590:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616563072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076450.793596:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076450.793598:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076450.793600:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616563072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076450.793604:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616563200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076450.793615:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076450.793621:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076450/real 1769076450] req@ffff995b868c4700 x1855007616563200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076450.793628:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616563200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076450.793633:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076450.793635:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076450.793637:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616563200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076451.496539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076451.496601:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076451.816557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076451.816558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076451.816567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076451.816587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076452.520627:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076452.520719:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076452.840570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076452.840572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076452.840578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076452.840589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076453.544543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076453.544626:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076453.864520:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076453.864527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076453.864566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076453.864587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076454.568576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076454.568696:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076454.888514:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076454.888539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076454.888540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076454.888555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076455.592548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076455.592641:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076455.912590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076455.912603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076455.912614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076455.912621:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076455.912635:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3441 00000100:00080000:1.0:1769076455.912649:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076455.912659:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076455.912678:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076455.912721:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616563328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076455.912724:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076455.912749:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076455.912753:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076455.912755:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076455.912758:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3441 00000100:00080000:1.0:1769076455.912772:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076455.912776:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076455.912784:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076455.912798:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616563456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076455.912806:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076455.912806:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076455.912812:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076455.912814:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076455.912818:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3441 00000100:00080000:1.0:1769076455.912821:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076455.912823:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076455.912835:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076455.912844:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616563584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076455.912849:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616563328:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076455.912862:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076455.912865:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076455.912867:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076455.912871:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3441 00000100:00080000:1.0:1769076455.912876:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076455.912879:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076455.912883:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076455.912893:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616563712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076455.912899:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076455.912901:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076455.912903:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076455.912906:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3441 00000100:00080000:1.0:1769076455.912909:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076455.912911:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076455.912915:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076455.912921:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076455.912923:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616563840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076455.912931:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076455.912934:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076455.912935:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076455.912939:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3441 00000100:00080000:1.0:1769076455.912944:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076455.912947:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076455.912949:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076455.912956:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616563968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076455.912963:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076455.912966:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076455.912967:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076455.912973:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3441 00000100:00080000:1.0:1769076455.912992:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076455.912997:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076455.913003:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076455.913014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076455/real 1769076455] req@ffff995b868c6300 x1855007616563328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076455.913017:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616564096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076455.913114:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616563328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076455.913139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076455.913148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076455.913159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616563328:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076455.913178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616563456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076455.913230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076455.913266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076455/real 1769076455] req@ffff995b868c7100 x1855007616563456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076455.913290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616563456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076455.913302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076455.913307:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076455.913313:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616563456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076455.913359:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616563584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076455.913388:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076455.913407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076455/real 1769076455] req@ffff995b868c6a00 x1855007616563584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076455.913513:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616563584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076455.913531:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076455.913536:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076455.913541:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616563584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076455.913556:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616563712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076455.913598:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076455.913621:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076455/real 1769076455] req@ffff995b868c4700 x1855007616563712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076455.913638:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616563712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076455.913650:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076455.913654:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076455.913658:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616563712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076455.913664:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616563840:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076455.913706:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076455.913713:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076455/real 1769076455] req@ffff995b868c7480 x1855007616563840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076455.913721:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616563840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076455.913727:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076455.913729:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076455.913731:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616563840:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076455.913734:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616563968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076455.913745:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076455.913762:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076455/real 1769076455] req@ffff995b868c5c00 x1855007616563968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076455.913769:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616563968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076455.913775:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076455.913778:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076455.913780:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616563968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076455.913783:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616564096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076455.913793:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076455.913799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076455/real 1769076455] req@ffff995b868c6d80 x1855007616564096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076455.913806:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616564096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076455.913811:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076455.913813:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076455.913815:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616564096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076456.616511:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076456.616622:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076456.936592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076456.936606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076456.936608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076456.936625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076457.640575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076457.640759:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076457.960588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076457.960589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076457.960656:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076457.960657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076458.664540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076458.664683:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076458.984562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076458.984565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076458.984605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076458.984607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076459.688536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076459.688639:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076460.008650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076460.008650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076460.008654:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076460.008679:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076460.712521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076460.712589:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076461.032561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076461.032566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076461.032578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076461.032609:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076461.032613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076461.032617:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076461.032625:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076461.032637:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3446 00000100:00080000:2.0:1769076461.032651:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076461.032662:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076461.032710:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616564224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076461.032751:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076461.032757:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076461.032769:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076461.032775:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3446 00000100:00080000:2.0:1769076461.032781:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076461.032788:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076461.032799:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616564224:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769076461.032800:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076461.032822:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616564352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076461.032837:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076461.032841:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076461.032843:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076461.032848:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3446 00000100:00080000:2.0:1769076461.032852:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076461.032856:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076461.032861:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076461.032874:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616564480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076461.032884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076461.032888:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076461.032891:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076461.032896:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076461.032902:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3446 00000100:00080000:2.0:1769076461.032909:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076461.032914:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076461.032922:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076461.032935:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616564608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076461.032946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076461.032951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076461/real 1769076461] req@ffff995b85ac9880 x1855007616564224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076461.032952:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076461.032958:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076461.032971:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3446 00000100:00080000:2.0:1769076461.032980:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076461.032990:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076461.032998:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076461.033007:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616564224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076461.033015:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616564736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076461.033024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076461.033037:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076461.033047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616564224:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076461.033049:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076461.033060:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076461.033064:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076461.033073:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3446 00000100:00100000:0.0:1769076461.033079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616564352:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076461.033080:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076461.033097:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076461.033107:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076461.033132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076461.033150:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616564864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076461.033160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076461/real 1769076461] req@ffff995b85acaa00 x1855007616564352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076461.033172:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076461.033176:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076461.033182:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076461.033195:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3446 00000100:00080000:2.0:1769076461.033206:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076461.033214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616564352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076461.033216:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076461.033229:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076461.033260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076461.033270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076461.033271:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616564992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076461.033278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616564352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076461.033298:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616564480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076461.033325:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076461.033367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076461/real 1769076461] req@ffff995b85ac9c00 x1855007616564480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076461.033382:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616564480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076461.033398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076461.033402:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076461.033405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616564480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076461.033412:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616564608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076461.033433:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076461.033443:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076461/real 1769076461] req@ffff995b85acb800 x1855007616564608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076461.033492:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616564608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076461.033507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076461.033513:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076461.033519:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616564608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076461.033529:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616564736:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076461.033563:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076461.033581:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076461/real 1769076461] req@ffff995b85aca300 x1855007616564736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076461.033594:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616564736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076461.033603:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076461.033606:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076461.033610:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616564736:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076461.033632:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616564864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076461.033650:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076461.033660:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076461/real 1769076461] req@ffff995b85ac8a80 x1855007616564864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076461.033672:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616564864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076461.033680:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076461.033683:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076461.033686:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616564864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076461.033691:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616564992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076461.033705:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076461.033714:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076461/real 1769076461] req@ffff995b85acb100 x1855007616564992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076461.033725:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616564992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076461.033733:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076461.033736:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076461.033739:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616564992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076461.736560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076461.736697:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076462.056577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076462.056581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076462.056620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076462.056625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076462.760556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076462.760691:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076463.080581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076463.080607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076463.080655:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076463.080656:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076463.784565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076463.784686:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076464.104616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076464.104624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076464.104636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076464.104694:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076464.808556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076464.808714:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076465.128587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076465.128605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076465.128642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076465.128654:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076465.832533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076465.832678:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076466.152553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076466.152555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076466.152562:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076466.152565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076466.152570:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076466.152577:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076466.152589:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3451 00000100:00100000:2.0:1769076466.152599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076466.152602:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076466.152613:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076466.152655:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616565120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076466.152680:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076466.152683:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076466.152685:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076466.152688:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3451 00000100:00080000:3.0:1769076466.152692:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076466.152697:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076466.152705:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076466.152717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616565120:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076466.152720:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616565248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076466.152732:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076466.152735:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076466.152736:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076466.152739:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3451 00000100:00080000:3.0:1769076466.152742:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076466.152745:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076466.152749:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076466.152760:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616565376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076466.152767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076466.152770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076466.152771:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076466.152775:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3451 00000100:00080000:3.0:1769076466.152778:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076466.152781:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076466.152785:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076466.152795:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616565504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076466.152806:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076466.152811:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076466.152813:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076466.152819:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3451 00000100:00100000:0.0:1769076466.152824:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616565248:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076466.152825:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076466.152835:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076466.152841:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076466.152857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616565376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076466.152859:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616565632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076466.152876:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076466.152880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616565504:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076466.152883:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076466.152888:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076466.152895:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3451 00000100:00080000:3.0:1769076466.152901:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076466.152907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616565632:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076466.152909:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076466.152919:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076466.152936:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616565760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076466.152961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076466.152966:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076466.152970:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076466.152975:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3451 00000100:00080000:3.0:1769076466.152980:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076466.152985:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076466.152992:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616565760:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076466.152993:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076466.153014:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616565888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076466.153037:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616565888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076466.856531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076466.856659:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076467.176590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076467.176598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076467.176638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076467.176725:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076467.880571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076467.880707:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076468.200605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076468.200618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076468.200666:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076468.200724:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076468.904607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076468.904762:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076469.224606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076469.224609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076469.224624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076469.224800:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:2.0:1769076469.224993:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769076469.225066:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076469.225096:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076469.225173:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076469.225186:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076469.225233:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076469.225261:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076469.225286:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076469.225291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076466/real 1769076469] req@ffff995b84558000 x1855007616565248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1769076469.225316:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076469.225350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616565248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076469.225374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076469.225387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076469.225396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616565248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076469.225414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076466/real 1769076469] req@ffff995b8455bb80 x1855007616565376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076469.225488:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616565376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076469.225512:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076469.225521:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076469.225528:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616565376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076469.225542:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076466/real 1769076469] req@ffff995b8455aa00 x1855007616565504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076469.225567:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616565504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076469.225583:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076469.225590:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076469.225597:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616565504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076469.225610:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076466/real 1769076469] req@ffff995b8455b100 x1855007616565120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076469.225637:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616565120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076469.225653:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076469.225661:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076469.225668:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616565120:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076469.225724:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076466/real 1769076469] req@ffff995b8455a680 x1855007616565888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076469.225753:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616565888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076469.225772:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076469.225782:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076469.225790:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616565888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076469.225807:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076466/real 1769076469] req@ffff995b84558a80 x1855007616565760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076469.225834:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616565760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076469.225853:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076469.225867:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076469.225876:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616565760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076469.225896:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076466/real 1769076469] req@ffff995b84559500 x1855007616565632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076469.225921:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616565632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076469.225939:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076469.225945:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076469.225953:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616565632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076469.928619:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076469.928710:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076470.248633:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076470.248656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076470.248704:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076470.248764:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076470.952607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076470.952691:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076471.272615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076471.272629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076471.272660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076471.272700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076471.272702:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076471.272725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076471.272732:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076471.272745:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3456 00000100:00080000:1.0:1769076471.272760:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076471.272770:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076471.272812:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616566016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076471.272846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076471.272851:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076471.272853:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076471.272859:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3456 00000100:00080000:1.0:1769076471.272862:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076471.272867:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076471.272880:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076471.272906:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616566144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076471.272931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616566016:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076471.272931:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076471.272946:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076471.272955:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076471.272966:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3456 00000100:00080000:1.0:1769076471.272973:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076471.272985:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076471.272997:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076471.273027:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616566272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076471.273046:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076471.273054:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076471.273058:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076471.273068:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076471.273077:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3456 00000100:00080000:1.0:1769076471.273089:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076471.273099:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076471.273110:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076471.273139:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616566400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076471.273150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076471/real 1769076471] req@ffff995b868c5500 x1855007616566016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076471.273234:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076471.273239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616566016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076471.273246:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076471.273254:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076471.273274:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3456 00000100:00080000:0.0:1769076471.273287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076471.273289:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076471.273309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076471.273312:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076471.273329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616566016:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076471.273329:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076471.273369:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616566528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076471.273392:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616566144:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076471.273392:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076471.273410:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076471.273417:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076471.273427:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3456 00000100:00080000:1.0:1769076471.273437:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076471.273509:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076471.273532:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076471.273546:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076471/real 1769076471] req@ffff995b868c7800 x1855007616566144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076471.273552:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076471.273594:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616566656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076471.273596:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616566144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076471.273658:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076471.273659:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076471.273679:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076471.273679:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076471.273687:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076471.273698:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616566144:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076471.273708:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3456 00000100:00080000:1.0:1769076471.273722:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076471.273730:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616566272:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076471.273740:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076471.273760:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076471.273792:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616566784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076471.273814:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076471.273848:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076471/real 1769076471] req@ffff995b868c4a80 x1855007616566272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076471.273876:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616566272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076471.273896:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076471.273905:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076471.273913:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616566272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076471.273927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616566400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076471.273970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076471.273992:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076471/real 1769076471] req@ffff995b868c6d80 x1855007616566400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076471.274016:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616566400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076471.274035:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076471.274041:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076471.274047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616566400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076471.274083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616566528:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076471.274125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076471.274152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076471/real 1769076471] req@ffff995b868c5c00 x1855007616566528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076471.274212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616566528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076471.274230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076471.274237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076471.274243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616566528:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076471.274255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616566656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076471.274296:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076471.274334:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076471/real 1769076471] req@ffff995b868c7480 x1855007616566656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076471.274357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616566656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076471.274373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076471.274380:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076471.274386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616566656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076471.274398:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616566784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076471.274437:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076471.274501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076471/real 1769076471] req@ffff995b868c4700 x1855007616566784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076471.274529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616566784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076471.274559:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076471.274566:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076471.274573:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616566784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076471.976587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076471.976655:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076472.296587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076472.296591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076472.296642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076472.296685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076473.000540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076473.000752:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076473.320586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076473.320586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076473.320586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076473.320651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076474.024595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076474.024698:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076474.344638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076474.344645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076474.344665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076474.344681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076475.048594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076475.048641:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076475.368632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076475.368640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076475.368692:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076475.368724:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076476.072571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076476.072662:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076476.392614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076476.392620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076476.392682:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076476.392693:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076476.392700:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076476.392715:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076476.392719:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3462 00000100:00080000:1.0:1769076476.392740:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076476.392754:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076476.392757:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076476.392825:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616566912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076476.392871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076476.392877:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076476.392880:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076476.392885:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3462 00000100:00080000:1.0:1769076476.392891:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076476.392900:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076476.392913:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076476.392936:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616567040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076476.392947:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616566912:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076476.392986:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076476.392990:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076476.392992:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076476.393000:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3462 00000100:00080000:1.0:1769076476.393008:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076476.393014:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076476.393020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076476.393035:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616567168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076476.393047:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076476.393052:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076476.393055:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076476.393060:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3462 00000100:00080000:1.0:1769076476.393126:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076476.393134:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076476.393142:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076476.393156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076476.393168:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616567296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076476.393184:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076476.393188:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076476.393193:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076476.393198:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3462 00000100:00080000:1.0:1769076476.393206:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076476.393211:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076476.393218:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076476.393237:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616567424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076476.393249:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076476.393261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076476/real 1769076476] req@ffff995b868c6a00 x1855007616566912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076476.393263:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076476.393274:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076476.393297:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3462 00000100:00080000:1.0:1769076476.393320:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076476.393331:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076476.393338:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076476.393344:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616566912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076476.393367:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616567552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076476.393373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076476.393396:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076476.393404:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076476.393417:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616566912:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076476.393420:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076476.393433:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076476.393445:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3462 00000100:00080000:1.0:1769076476.393487:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076476.393493:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076476.393502:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076476.393528:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616567680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076476.393550:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076476.393600:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076476.393620:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076476/real 1769076476] req@ffff995b868c7100 x1855007616567040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076476.393637:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616567040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076476.393650:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076476.393655:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076476.393660:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076476.393668:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076476.393689:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076476.393701:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076476/real 1769076476] req@ffff995b868c6300 x1855007616567168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076476.393716:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616567168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076476.393730:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076476.393734:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076476.393738:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076476.393745:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567296:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076476.393766:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076476.393777:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076476/real 1769076476] req@ffff995b868c4700 x1855007616567296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076476.393792:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616567296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076476.393803:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076476.393806:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076476.393811:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076476.393818:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567424:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076476.393836:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076476.393847:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076476/real 1769076476] req@ffff995b868c7480 x1855007616567424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076476.393860:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616567424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076476.393871:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076476.393875:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076476.393879:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567424:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076476.393886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616567552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076476.393904:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076476.393915:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076476/real 1769076476] req@ffff995b868c5c00 x1855007616567552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076476.393928:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616567552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076476.393939:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076476.393943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076476.393947:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616567552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076476.393972:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616567680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076476.393991:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076476.394005:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076476/real 1769076476] req@ffff995b868c6d80 x1855007616567680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076476.394020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616567680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076476.394038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076476.394045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076476.394052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616567680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076477.096580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076477.096767:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076477.416587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076477.416601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076477.416692:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076477.416725:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076478.120560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076478.120699:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076478.440746:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076478.440768:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076478.440800:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076478.440908:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076479.144565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076479.144677:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076479.464566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076479.464577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076479.464593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076479.464652:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076480.168620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076480.168648:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076480.488558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076480.488609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076480.488614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076480.488645:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076481.192561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076481.192694:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076481.512606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076481.512615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076481.512616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076481.512623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076481.512637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076481.512646:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076481.512660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076481.512662:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3467 00000100:00080000:2.0:1769076481.512681:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076481.512696:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076481.512750:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616567808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076481.512807:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076481.512812:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076481.512815:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076481.512822:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3467 00000100:00080000:2.0:1769076481.512829:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076481.512836:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076481.512850:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076481.512876:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616567808:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076481.512876:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616567936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076481.512905:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076481.512909:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076481.512911:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076481.512916:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3467 00000100:00080000:2.0:1769076481.512921:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076481.512925:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076481.512932:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076481.512948:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616568064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076481.512961:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076481.512965:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076481.512968:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076481.512973:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3467 00000100:00080000:2.0:1769076481.512979:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076481.512983:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076481.513011:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076481.513021:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076481.513031:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616568192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076481.513045:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076481.513049:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076481.513052:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076481.513056:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3467 00000100:00080000:2.0:1769076481.513061:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076481.513066:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076481.513073:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076481.513082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076481/real 1769076481] req@ffff995b85ac8380 x1855007616567808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076481.513092:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616568320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076481.513127:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076481.513132:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769076481.513138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616567808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076481.513139:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076481.513152:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3467 00000100:00080000:2.0:1769076481.513162:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076481.513173:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076481.513176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076481.513181:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076481.513188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076481.513197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616567808:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076481.513202:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616568448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076481.513217:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076481.513221:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076481.513223:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076481.513230:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3467 00000100:00100000:0.0:1769076481.513237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567936:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076481.513250:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076481.513257:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076481.513264:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076481.513280:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616568576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076481.513289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076481.513335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076481/real 1769076481] req@ffff995b85acbb80 x1855007616567936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076481.513355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616567936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076481.513366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076481.513371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076481.513376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616567936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076481.513383:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076481.513406:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076481.513423:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076481/real 1769076481] req@ffff995b85acb100 x1855007616568064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076481.513448:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616568064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076481.513516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076481.513525:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076481.513531:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076481.513541:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076481.513582:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076481.513605:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076481/real 1769076481] req@ffff995b85ac8a80 x1855007616568192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076481.513631:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616568192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076481.513651:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076481.513658:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076481.513666:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076481.513679:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568320:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076481.513710:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076481.513726:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076481/real 1769076481] req@ffff995b85aca300 x1855007616568320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076481.513741:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616568320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076481.513753:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076481.513757:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076481.513761:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568320:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076481.513768:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616568448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076481.513787:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076481.513799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076481/real 1769076481] req@ffff995b85acb800 x1855007616568448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076481.513813:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616568448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076481.513831:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076481.513835:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076481.513841:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616568448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076481.513880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616568576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076481.513920:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076481.513941:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076481/real 1769076481] req@ffff995b85ac9c00 x1855007616568576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076481.513957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616568576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076481.513968:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076481.513972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076481.513976:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616568576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076482.216575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076482.216710:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076482.536614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076482.536627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076482.536675:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076482.536686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076483.240573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076483.240715:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076483.560607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076483.560613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076483.560626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076483.560664:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076484.264542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076484.264735:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076484.584575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076484.584669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076484.584679:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076484.584749:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076485.288549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076485.288751:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076485.608609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076485.608621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076485.608658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076485.608696:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076486.312573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076486.312628:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076486.632570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076486.632597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076486.632608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076486.632618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076486.632706:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076486.632715:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076486.632720:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076486.632734:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3472 00000100:00080000:0.0:1769076486.632749:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076486.632759:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076486.632808:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616568704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076486.632848:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076486.632855:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076486.632859:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076486.632867:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3472 00000100:00080000:0.0:1769076486.632874:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076486.632883:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076486.632899:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076486.632969:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616568832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076486.632969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616568704:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076486.632990:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076486.632996:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076486.632999:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076486.633006:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3472 00000100:00080000:0.0:1769076486.633013:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076486.633020:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076486.633029:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076486.633046:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616568960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076486.633063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769076486.633065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076486.633069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076486.633072:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076486.633079:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3472 00000100:00080000:0.0:1769076486.633085:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076486.633092:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076486.633101:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076486.633118:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616569088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076486.633123:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076486/real 1769076486] req@ffff995b84643800 x1855007616568704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076486.633137:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076486.633156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616568704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076486.633161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076486.633165:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076486.633172:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3472 00000100:00080000:2.0:1769076486.633177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076486.633178:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076486.633185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076486.633186:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076486.633194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616568704:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076486.633195:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076486.633214:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616569216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076486.633230:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568832:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076486.633231:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076486.633238:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076486.633242:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076486.633249:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3472 00000100:00080000:0.0:1769076486.633255:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076486.633262:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076486.633270:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076486.633271:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076486.633288:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616569344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076486.633294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076486/real 1769076486] req@ffff995b84642a00 x1855007616568832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076486.633308:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076486.633315:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076486.633319:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076486.633321:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616568832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076486.633326:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3472 00000100:00080000:0.0:1769076486.633332:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076486.633339:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076486.633339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076486.633346:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769076486.633348:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076486.633353:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076486.633363:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616569472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076486.633365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076486.633399:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076486.633423:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076486/real 1769076486] req@ffff995b84640e00 x1855007616568960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076486.633450:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616568960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076486.633505:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076486.633510:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076486.633515:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616568960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076486.633523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076486.633555:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076486.633571:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076486/real 1769076486] req@ffff995b84640000 x1855007616569088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076486.633585:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616569088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076486.633597:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076486.633601:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076486.633605:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076486.633627:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569216:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076486.633645:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076486.633657:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076486/real 1769076486] req@ffff995b84641c00 x1855007616569216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076486.633670:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616569216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076486.633681:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076486.633684:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076486.633689:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569216:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076486.633696:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616569344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076486.633713:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076486.633724:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076486/real 1769076486] req@ffff995b84642680 x1855007616569344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076486.633737:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616569344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076486.633748:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076486.633751:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076486.633755:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616569344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076486.633761:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616569472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076486.633778:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076486.633789:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076486/real 1769076486] req@ffff995b84640700 x1855007616569472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076486.633803:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616569472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076486.633813:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076486.633816:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076486.633819:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616569472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076487.336594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076487.336750:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076487.656589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076487.656616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076487.656625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076487.656633:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076488.360664:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076488.360678:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076488.680601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076488.680653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076488.680677:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076488.680706:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076489.384603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076489.384669:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076489.704604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076489.704651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076489.704654:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076489.704661:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076490.408527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076490.408603:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076490.728519:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076490.728538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076490.728556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076490.728558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076491.432517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076491.432587:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076491.752560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076491.752561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076491.752562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076491.752614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076491.752622:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076491.752628:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076491.752632:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076491.752640:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3477 00000100:00080000:0.0:1769076491.752650:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076491.752655:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076491.752685:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616569600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076491.752708:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076491.752712:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076491.752714:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076491.752717:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3477 00000100:00080000:0.0:1769076491.752720:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076491.752724:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076491.752732:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076491.752744:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616569728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076491.752751:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616569600:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076491.752757:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076491.752760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076491.752761:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076491.752766:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3477 00000100:00080000:0.0:1769076491.752768:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076491.752771:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076491.752775:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076491.752785:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616569856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076491.752792:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076491.752795:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076491.752797:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076491.752801:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3477 00000100:00080000:0.0:1769076491.752805:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076491.752809:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076491.752815:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076491.752872:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616569984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076491.752882:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076491.752885:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076491.752886:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076491.752889:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3477 00000100:00080000:0.0:1769076491.752893:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076491.752897:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076491.752901:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076491.752910:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616570112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076491.752917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076491.752920:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076491.752922:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076491.752927:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3477 00000100:00080000:0.0:1769076491.752930:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076491.752935:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076491.752940:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076491.752950:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076491.752951:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616570240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076491.752963:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076491.752967:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076491.752969:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076491.752974:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3477 00000100:00080000:0.0:1769076491.752979:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076491.752983:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076491.752989:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076491.752998:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616570368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076491.753080:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076491/real 1769076491] req@ffff995b84640380 x1855007616569600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076491.753097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616569600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076491.753106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076491.753110:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076491.753114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616569600:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076491.753183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076491.753206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076491.753217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076491/real 1769076491] req@ffff995b84641f80 x1855007616569728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076491.753225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616569728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076491.753231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076491.753234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076491.753237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076491.753241:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076491.753251:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076491.753258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076491/real 1769076491] req@ffff995b84640a80 x1855007616569856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076491.753265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616569856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076491.753272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076491.753274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076491.753275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076491.753279:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569984:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076491.753290:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076491.753305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076491/real 1769076491] req@ffff995b84640700 x1855007616569984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076491.753313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616569984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076491.753319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076491.753321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076491.753323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616569984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076491.753327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616570112:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076491.753338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076491.753344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076491/real 1769076491] req@ffff995b84642680 x1855007616570112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076491.753352:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616570112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076491.753358:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076491.753359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076491.753361:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616570112:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076491.753365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616570240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076491.753374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076491.753380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076491/real 1769076491] req@ffff995b84641c00 x1855007616570240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076491.753387:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616570240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076491.753392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076491.753394:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076491.753396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616570240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076491.753399:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616570368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076491.753408:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076491.753414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076491/real 1769076491] req@ffff995b84640000 x1855007616570368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076491.753426:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616570368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076491.753431:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076491.753433:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076491.753435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616570368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076492.456533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076492.456626:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076492.776559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076492.776559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076492.776561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076492.776593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769076493.480622:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076493.480635:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076493.800566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076493.800567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076493.800576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076493.800601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076494.504535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076494.504635:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076494.824584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076494.824585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076494.824604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076494.824650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076495.528539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076495.528589:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076495.848512:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076495.848565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076495.848569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076495.848669:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076496.552607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076496.552661:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076496.872681:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076496.872688:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076496.872688:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076496.872703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076496.872713:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076496.872758:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3482 00000100:00080000:2.0:1769076496.872781:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076496.872786:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076496.872792:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076496.872799:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076496.872889:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616570496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076496.872936:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076496.872944:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076496.872948:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076496.872957:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3482 00000100:00080000:2.0:1769076496.872964:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076496.872973:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076496.872990:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076496.873007:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616570496:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076496.873014:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616570624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076496.873031:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076496.873037:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076496.873040:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076496.873046:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3482 00000100:00080000:2.0:1769076496.873052:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076496.873058:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076496.873065:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076496.873083:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616570752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076496.873098:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076496.873115:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076496.873119:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076496.873123:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076496.873129:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3482 00000100:00080000:2.0:1769076496.873135:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076496.873142:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076496.873150:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076496.873167:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616570880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076496.873181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076496/real 1769076496] req@ffff995b85acaa00 x1855007616570496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076496.873183:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076496.873190:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076496.873194:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076496.873200:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3482 00000100:00080000:2.0:1769076496.873205:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076496.873212:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076496.873213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616570496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076496.873220:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769076496.873232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076496.873235:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616571008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076496.873240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076496.873247:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616570496:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076496.873252:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076496.873258:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076496.873261:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076496.873267:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3482 00000100:00080000:2.0:1769076496.873272:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076496.873277:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076496.873284:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076496.873286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616570624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076496.873300:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616571136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076496.873317:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076496.873323:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1769076496.873323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076496.873327:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076496.873333:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3482 00000100:00080000:2.0:1769076496.873340:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076496.873346:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076496.873346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076496/real 1769076496] req@ffff995b85ac9880 x1855007616570624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076496.873356:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076496.873369:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616570624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076496.873387:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076496.873390:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616571264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076496.873393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076496.873400:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616570624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076496.873411:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616570752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076496.873447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076496.873505:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076496/real 1769076496] req@ffff995b85ac9c00 x1855007616570752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076496.873522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616570752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076496.873535:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076496.873539:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076496.873543:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616570752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076496.873551:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616570880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076496.873577:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076496.873589:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076496/real 1769076496] req@ffff995b85acb800 x1855007616570880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076496.873602:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616570880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076496.873614:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076496.873617:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076496.873621:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616570880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076496.873627:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571008:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076496.873643:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076496.873654:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076496/real 1769076496] req@ffff995b85aca300 x1855007616571008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076496.873666:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616571008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076496.873677:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076496.873680:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076496.873683:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076496.873703:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616571136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076496.873721:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076496.873756:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076496/real 1769076496] req@ffff995b85ac8a80 x1855007616571136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076496.873772:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616571136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076496.873782:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076496.873785:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076496.873790:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616571136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076496.873798:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616571264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076496.873818:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076496.873830:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076496/real 1769076496] req@ffff995b85acb100 x1855007616571264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076496.873843:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616571264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076496.873853:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076496.873856:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076496.873860:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616571264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076497.576541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076497.576628:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076497.896557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076497.896580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076497.896592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076497.896593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076498.600554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076498.600656:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076498.920616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076498.920616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076498.920693:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076498.920731:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076499.624506:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076499.624606:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076499.944537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076499.944537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076499.944541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076499.944593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076500.648534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076500.648634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076500.968546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076500.968557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076500.968575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076500.968596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076501.672528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076501.672576:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076501.992572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076501.992572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076501.992578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076501.992581:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076501.992588:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076501.992593:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076501.992604:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3487 00000100:00080000:2.0:1769076501.992615:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076501.992624:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076501.992629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076501.992692:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616571392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076501.992715:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076501.992719:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076501.992721:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076501.992726:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3487 00000100:00080000:2.0:1769076501.992730:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076501.992735:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076501.992754:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076501.992758:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616571392:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076501.992769:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616571520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076501.992779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076501.992782:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076501.992784:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076501.992787:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3487 00000100:00080000:2.0:1769076501.992790:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076501.992793:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076501.992798:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076501.992807:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616571648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076501.992816:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076501.992819:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1769076501.992821:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076501.992822:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076501.992825:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3487 00000100:00080000:2.0:1769076501.992828:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076501.992831:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076501.992845:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076501.992854:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616571776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076501.992862:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076501.992865:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076501.992867:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076501.992870:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076501/real 1769076501] req@ffff995b85acbb80 x1855007616571392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076501.992871:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3487 00000100:00080000:2.0:1769076501.992874:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076501.992878:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076501.992882:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076501.992888:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616571392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076501.992890:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616571904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076501.992899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076501.992900:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076501.992902:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076501.992904:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076501.992905:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076501.992909:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3487 00000100:00100000:3.0:1769076501.992909:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616571392:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076501.992912:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076501.992916:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076501.992930:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076501.992937:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616572032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076501.992943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571520:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076501.992957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076501.992961:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076501.992963:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769076501.992964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076501.992967:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3487 00000100:00080000:2.0:1769076501.992970:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076501.992974:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076501.992977:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076501/real 1769076501] req@ffff995b85ac8380 x1855007616571520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076501.992979:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076501.992999:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616572160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076501.993000:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616571520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.993009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.993012:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076501.993016:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076501.993022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076501.993041:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076501.993053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076501/real 1769076501] req@ffff995b85acb100 x1855007616571648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.993062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616571648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.993080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.993082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076501.993085:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076501.993089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571776:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076501.993102:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076501.993109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076501/real 1769076501] req@ffff995b85ac8a80 x1855007616571776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.993117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616571776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.993122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.993124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076501.993126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076501.993130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571904:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076501.993139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076501.993144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076501/real 1769076501] req@ffff995b85aca300 x1855007616571904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.993151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616571904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.993157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.993159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076501.993160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616571904:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076501.993171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616572032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076501.993186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076501.993192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076501/real 1769076501] req@ffff995b85acb800 x1855007616572032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.993199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616572032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.993205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.993207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076501.993209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616572032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076501.993212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616572160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076501.993221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076501.993227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076501/real 1769076501] req@ffff995b85ac9c00 x1855007616572160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.993234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616572160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.993239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.993240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076501.993242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616572160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076502.696528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076502.696600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076503.016560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076503.016562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076503.016574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076503.016595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076503.720533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076503.720600:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076504.040549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076504.040555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076504.040575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076504.040583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076504.744527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076504.744581:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076505.064518:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076505.064535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076505.064564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076505.064691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076505.768525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076505.768615:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076506.088524:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076506.088555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076506.088564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076506.088589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076506.792545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076506.792592:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076507.112534:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076507.112569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076507.112585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076507.112592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076507.112628:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076507.112634:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076507.112638:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076507.112646:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3492 00000100:00080000:1.0:1769076507.112657:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076507.112663:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076507.112698:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616572288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076507.112727:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076507.112732:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076507.112734:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076507.112741:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3492 00000100:00080000:1.0:1769076507.112746:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076507.112751:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076507.112760:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076507.112768:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616572288:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076507.112775:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616572416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076507.112785:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076507.112788:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076507.112790:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076507.112795:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3492 00000100:00080000:1.0:1769076507.112798:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076507.112800:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076507.112805:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076507.112813:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616572544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076507.112820:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076507.112822:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076507.112824:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076507.112828:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3492 00000100:00080000:1.0:1769076507.112830:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076507.112832:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076507.112835:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076507.112845:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616572672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076507.112852:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076507.112854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1769076507.112859:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076507.112860:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076507.112867:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3492 00000100:00080000:1.0:1769076507.112870:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076507.112873:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076507.112878:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076507.112888:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616572800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076507.112895:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076507.112897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076507.112898:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076507.112901:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3492 00000100:00080000:1.0:1769076507.112903:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076507.112906:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076507.112913:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076507.112928:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616572928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076507.112929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076507/real 1769076507] req@ffff995b868c4a80 x1855007616572288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076507.112991:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076507.112994:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076507.112996:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076507.113000:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3492 00000100:00080000:1.0:1769076507.113003:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076507.113008:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616572288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076507.113011:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076507.113024:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076507.113032:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076507.113037:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616573056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076507.113039:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076507.113056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616572288:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076507.113098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616572416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076507.113125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076507.113141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076507/real 1769076507] req@ffff995b868c7800 x1855007616572416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076507.113156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616572416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076507.113166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076507.113170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076507.113172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616572416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076507.113178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616572544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076507.113195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076507.113204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076507/real 1769076507] req@ffff995b868c5500 x1855007616572544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076507.113212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616572544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076507.113225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076507.113228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076507.113230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616572544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076507.113235:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616572672:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076507.113247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076507.113253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076507/real 1769076507] req@ffff995b868c6d80 x1855007616572672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076507.113260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616572672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076507.113267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076507.113268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076507.113270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616572672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076507.113275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616572800:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076507.113285:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076507.113291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076507/real 1769076507] req@ffff995b868c5c00 x1855007616572800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076507.113298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616572800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076507.113305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076507.113306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076507.113309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616572800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076507.113313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616572928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076507.113323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076507.113329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076507/real 1769076507] req@ffff995b868c7480 x1855007616572928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076507.113336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616572928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076507.113354:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076507.113358:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076507.113362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616572928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076507.113369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616573056:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076507.113392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076507.113405:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076507/real 1769076507] req@ffff995b868c4700 x1855007616573056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076507.113415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616573056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076507.113421:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076507.113424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076507.113426:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616573056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076507.816560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076507.816619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076508.136540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076508.136556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076508.136583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076508.136643:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076508.840527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076508.840600:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076509.160523:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076509.160546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076509.160567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076509.160586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076509.864529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076509.864656:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076510.184527:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076510.184538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076510.184553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076510.184556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076510.888558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076510.888603:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076511.208536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076511.208548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076511.208557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076511.208609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076511.912535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076511.912585:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076512.232592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076512.232604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076512.232621:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076512.232627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076512.232634:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076512.232643:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076512.232659:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3497 00000100:00100000:1.0:1769076512.232666:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076512.232688:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076512.232701:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076512.232767:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616573184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076512.232794:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076512.232798:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076512.232800:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076512.232804:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3497 00000100:00080000:0.0:1769076512.232807:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076512.232812:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076512.232832:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076512.232843:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616573312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076512.232852:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076512.232858:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076512.232861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616573184:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076512.232862:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076512.232877:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3497 00000100:00080000:0.0:1769076512.232894:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076512.232899:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076512.232905:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076512.232924:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616573440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076512.232937:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076512.232941:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076512.232944:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076512.232959:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3497 00000100:00080000:0.0:1769076512.232964:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076512.232967:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076512.232974:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076512.232985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076512.232987:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616573568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076512.232996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076512.233000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076512.233001:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076512.233007:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3497 00000100:00080000:0.0:1769076512.233022:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076512.233025:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076512.233031:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076512.233042:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616573696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076512.233053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076512/real 1769076512] req@ffff995b84640e00 x1855007616573184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076512.233056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076512.233073:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076512.233080:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076512.233092:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3497 00000100:00080000:0.0:1769076512.233097:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076512.233105:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076512.233106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616573184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076512.233119:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076512.233126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076512.233131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076512.233135:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616573824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076512.233141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616573184:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076512.233169:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076512.233172:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076512.233175:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076512.233180:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3497 00000100:00080000:0.0:1769076512.233183:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076512.233186:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076512.233194:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076512.233196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616573312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076512.233206:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616573952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076512.233224:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076512.233236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076512/real 1769076512] req@ffff995b84642a00 x1855007616573312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076512.233246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616573312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076512.233252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076512.233255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076512.233258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616573312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076512.233262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616573440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076512.233273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076512.233279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076512/real 1769076512] req@ffff995b84643800 x1855007616573440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076512.233286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616573440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076512.233293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076512.233296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076512.233297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616573440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076512.233301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616573568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076512.233311:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076512.233317:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076512/real 1769076512] req@ffff995b84641180 x1855007616573568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076512.233324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616573568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076512.233329:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076512.233331:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076512.233333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616573568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076512.233337:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616573696:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076512.233346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076512.233352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076512/real 1769076512] req@ffff995b84641500 x1855007616573696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076512.233363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616573696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076512.233369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076512.233371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076512.233373:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616573696:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076512.233377:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616573824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076512.233387:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076512.233392:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076512/real 1769076512] req@ffff995b84643b80 x1855007616573824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076512.233399:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616573824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076512.233405:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076512.233406:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076512.233408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616573824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076512.233422:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616573952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076512.233431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076512.233437:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076512/real 1769076512] req@ffff995b84641880 x1855007616573952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076512.233444:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616573952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076512.233453:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076512.233455:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076512.233457:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616573952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076512.936537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076512.936586:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076513.256560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076513.256567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076513.256596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076513.256598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076513.960524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076513.960574:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076514.280536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076514.280546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076514.280574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076514.280589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076514.984518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076514.984579:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076515.304540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076515.304541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076515.304570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076515.304669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076516.008565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076516.008631:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076516.328552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076516.328552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076516.328552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076516.328587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076517.032509:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076517.032595:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076517.352541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076517.352544:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076517.352560:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076517.352568:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076517.352574:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076517.352584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076517.352586:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3503 00000100:00080000:0.0:1769076517.352601:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076517.352603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076517.352610:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076517.352646:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616574080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076517.352668:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076517.352671:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076517.352673:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076517.352677:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3503 00000100:00080000:0.0:1769076517.352679:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076517.352682:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076517.352690:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076517.352699:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616574208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076517.352708:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076517.352713:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076517.352713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616574080:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076517.352716:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076517.352725:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3503 00000100:00080000:0.0:1769076517.352727:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076517.352729:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076517.352732:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076517.352743:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616574336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076517.352749:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076517.352752:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076517.352753:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076517.352756:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3503 00000100:00080000:0.0:1769076517.352759:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076517.352762:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076517.352768:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076517.352781:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616574464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076517.352792:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076517.352793:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076517.352800:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076517.352802:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076517.352807:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3503 00000100:00080000:0.0:1769076517.352813:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076517.352817:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076517.352823:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076517.352834:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616574592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076517.352840:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076517.352843:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076517/real 1769076517] req@ffff995b84643480 x1855007616574080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076517.352844:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076517.352847:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076517.352856:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3503 00000100:00080000:0.0:1769076517.352862:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076517.352867:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076517.352871:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076517.352877:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616574080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076517.352881:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616574720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076517.352907:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076517.352908:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076517.352912:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076517.352915:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076517.352918:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076517.352922:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3503 00000100:00080000:0.0:1769076517.352926:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076517.352927:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616574080:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076517.352932:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076517.352938:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076517.352945:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616574848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076517.352959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616574208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076517.352987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076517.353001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076517/real 1769076517] req@ffff995b84643100 x1855007616574208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076517.353020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616574208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076517.353029:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076517.353032:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076517.353035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616574208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076517.353041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616574336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076517.353060:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076517.353071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076517/real 1769076517] req@ffff995b84642d80 x1855007616574336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076517.353084:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616574336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076517.353095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076517.353099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076517.353102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616574336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076517.353109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616574464:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076517.353127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076517.353137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076517/real 1769076517] req@ffff995b84641880 x1855007616574464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076517.353150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616574464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076517.353165:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076517.353169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076517.353172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616574464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076517.353178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616574592:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076517.353194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076517.353204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076517/real 1769076517] req@ffff995b84643b80 x1855007616574592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076517.353215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616574592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076517.353225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076517.353228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076517.353232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616574592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076517.353238:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616574720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076517.353257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076517.353268:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076517/real 1769076517] req@ffff995b84641500 x1855007616574720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076517.353279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616574720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076517.353288:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076517.353292:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076517.353296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616574720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076517.353302:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616574848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076517.353319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076517.353330:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076517/real 1769076517] req@ffff995b84641180 x1855007616574848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076517.353342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616574848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076517.353350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076517.353353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076517.353357:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616574848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076518.056514:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076518.056577:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076518.376534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076518.376543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076518.376562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076518.376585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076519.080643:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076519.080739:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076519.400634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076519.400636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076519.400646:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076519.400678:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076520.104561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076520.104682:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076520.424616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076520.424662:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076520.424683:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076520.424737:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076521.128589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076521.128671:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076521.448569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076521.448624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076521.448648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076521.448668:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076522.152551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076522.152750:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076522.472540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076522.472540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076522.472552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076522.472557:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076522.472566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076522.472572:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076522.472578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076522.472583:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3508 00000100:00080000:2.0:1769076522.472597:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076522.472606:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076522.472646:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616574976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076522.472677:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076522.472682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076522.472684:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076522.472690:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3508 00000100:00080000:2.0:1769076522.472696:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076522.472701:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076522.472712:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076522.472713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616574976:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076522.472734:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616575104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076522.472750:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076522.472754:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076522.472757:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076522.472761:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3508 00000100:00080000:2.0:1769076522.472769:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076522.472774:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076522.472780:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076522.472786:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076522.472801:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616575232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076522.472814:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076522.472818:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076522.472821:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076522.472829:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076522/real 1769076522] req@ffff995b85ac9880 x1855007616574976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076522.472830:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3508 00000100:00080000:2.0:1769076522.472838:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076522.472847:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076522.472854:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076522.472855:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616574976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076522.472868:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076522.472874:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076522.472875:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616575360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076522.472880:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616574976:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076522.472898:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076522.472903:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076522.472906:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076522.472911:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616575104:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076522.472916:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3508 00000100:00080000:2.0:1769076522.472924:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076522.472930:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076522.472936:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076522.472938:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076522.472957:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616575488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076522.472958:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076522/real 1769076522] req@ffff995b85acaa00 x1855007616575104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076522.472989:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616575104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076522.473020:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076522.473025:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076522.473026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076522.473035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616575104:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076522.473036:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076522.473042:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076522.473047:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616575232:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076522.473052:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3508 00000100:00080000:2.0:1769076522.473060:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076522.473065:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769076522.473071:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769076522.473076:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076522.473083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076522/real 1769076522] req@ffff995b85ac9c00 x1855007616575232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076522.473095:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616575616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076522.473096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616575232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076522.473134:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076522.473138:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076522.473143:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076522.473146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076522.473151:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3508 00000100:00080000:3.0:1769076522.473155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076522.473160:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076522.473161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616575232:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076522.473170:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076522.473174:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616575360:192.168.204.132@tcp:8:kworker.0 00000008:00100000:2.0:1769076522.473180:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076522.473199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076522.473200:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616575744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076522.473213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076522/real 1769076522] req@ffff995b85acb800 x1855007616575360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076522.473225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616575360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076522.473234:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076522.473238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076522.473240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616575360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076522.473253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616575488:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076522.473265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076522.473272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076522/real 1769076522] req@ffff995b85aca300 x1855007616575488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076522.473281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616575488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076522.473288:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076522.473290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076522.473292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616575488:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076522.473296:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616575616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076522.473328:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076522.473336:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076522/real 1769076522] req@ffff995b85ac8a80 x1855007616575616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076522.473362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616575616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076522.473369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076522.473371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076522.473374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616575616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076522.473378:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616575744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076522.473390:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076522.473397:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076522/real 1769076522] req@ffff995b85acb100 x1855007616575744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076522.473405:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616575744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076522.473412:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076522.473413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076522.473415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616575744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076523.176609:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076523.176723:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076523.496587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076523.496605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076523.496656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076523.496668:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076524.200584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076524.200770:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076524.520599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076524.520629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076524.520643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076524.520661:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076525.224534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076525.224676:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076525.544628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076525.544628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076525.544628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076525.544689:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076526.248610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076526.248740:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076526.568592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076526.568631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076526.568631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076526.568715:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076527.272579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076527.272690:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076527.592647:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076527.592667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769076527.592667:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076527.592671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076527.592677:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076527.592706:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3513 00000100:00080000:1.0:1769076527.592716:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076527.592724:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076527.592724:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076527.592740:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076527.592761:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616575872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076527.592791:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076527.592796:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076527.592799:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076527.592805:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3513 00000100:00080000:1.0:1769076527.592810:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076527.592815:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076527.592820:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616575872:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076527.592827:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076527.592845:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616576000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076527.592856:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076527.592860:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076527.592862:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076527.592866:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3513 00000100:00080000:1.0:1769076527.592871:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076527.592874:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076527.592878:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076527.592887:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076527.592891:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616576128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076527.592901:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076527.592905:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076527.592906:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076527.592910:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3513 00000100:00080000:1.0:1769076527.592914:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076527.592918:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076527.592923:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076527.592934:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076527/real 1769076527] req@ffff995b868c6300 x1855007616575872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076527.592936:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616576256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076527.592969:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076527.592975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076527.592978:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076527.592982:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3513 00000100:00080000:1.0:1769076527.592984:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076527.592987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616575872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076527.592991:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076527.592997:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076527.593006:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616576384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076527.593012:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076527.593015:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076527.593018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076527.593020:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076527.593023:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3513 00000100:00080000:1.0:1769076527.593026:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076527.593029:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076527.593031:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076527.593036:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616576512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076527.593042:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076527.593044:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076527.593045:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076527.593047:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3513 00000100:00080000:1.0:1769076527.593049:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076527.593052:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076527.593055:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076527.593061:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616576640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076527.593076:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076527.593083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616575872:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076527.593157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076527.593211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076527.593287:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076527/real 1769076527] req@ffff995b868c7100 x1855007616576000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076527.593523:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616576000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076527.593533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076527.593629:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076527.593640:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076527.593657:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076527.593710:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076527.593725:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076527/real 1769076527] req@ffff995b868c6a00 x1855007616576128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076527.593736:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616576128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076527.593748:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076527.593751:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076527.593753:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076527.593758:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076527.593774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076527.593783:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076527/real 1769076527] req@ffff995b868c4700 x1855007616576256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076527.593796:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616576256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076527.593806:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076527.593809:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076527.593813:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076527.593820:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576384:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076527.593842:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076527.593856:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076527/real 1769076527] req@ffff995b868c7480 x1855007616576384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076527.593869:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616576384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076527.593877:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076527.593879:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076527.593882:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576384:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076527.593887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616576512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076527.593903:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076527.593910:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076527/real 1769076527] req@ffff995b868c5c00 x1855007616576512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076527.593919:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616576512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076527.593924:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076527.593926:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076527.593928:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616576512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076527.593932:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616576640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076527.593943:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076527.593951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076527/real 1769076527] req@ffff995b868c6d80 x1855007616576640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076527.593964:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616576640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076527.593973:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076527.593977:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076527.593981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616576640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076528.296519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076528.296568:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076528.616551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076528.616551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076528.616602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076528.616616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076529.320530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076529.320594:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076529.640553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076529.640596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076529.640601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076529.640627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076530.344666:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076530.344757:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076530.664599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076530.664616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076530.664624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076530.664641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076531.368581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076531.368763:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076531.688585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076531.688638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076531.688657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076531.688681:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076532.392593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076532.392659:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076532.712552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076532.712553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076532.712561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076532.712571:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076532.712578:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076532.712582:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076532.712591:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3518 00000100:00100000:0.0:1769076532.712601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076532.712601:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076532.712640:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076532.712674:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616576768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076532.712695:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076532.712698:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076532.712700:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076532.712703:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3518 00000100:00080000:1.0:1769076532.712706:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076532.712709:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076532.712717:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076532.712731:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616576896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076532.712743:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616576768:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076532.712755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076532.712760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076532.712762:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076532.712769:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3518 00000100:00080000:1.0:1769076532.712774:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076532.712779:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076532.712785:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076532.712814:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616577024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076532.712821:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076532.712823:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076532.712825:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076532.712828:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3518 00000100:00080000:1.0:1769076532.712831:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076532.712845:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076532.712850:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076532.712859:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616577152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076532.712883:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076532.712886:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076532.712887:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076532.712890:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576896:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076532.712894:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3518 00000100:00080000:1.0:1769076532.712900:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076532.712903:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076532.712908:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076532.712919:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616577280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076532.712928:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076532.712930:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076532.712932:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076532.712936:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577024:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076532.712938:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3518 00000100:00080000:1.0:1769076532.712947:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076532.712951:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076532.712966:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076532.712972:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076532.712984:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616577408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076532.712994:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577280:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769076532.713001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076532.713006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076532.713008:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076532.713013:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3518 00000100:00100000:3.0:1769076532.713017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616577408:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076532.713031:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076532.713037:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076532.713043:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076532.713056:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616577536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076532.713083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616577536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076533.416515:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076533.416579:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076533.736548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076533.736564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076533.736605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076533.736616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076534.440507:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076534.440565:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076534.760605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076534.760618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076534.760630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076534.760650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076535.464516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076535.464571:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076535.784535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076535.784536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076535.784538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076535.784682:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000400:00000100:1.0:1769076535.784699:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076535.784734:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076535.784750:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076535.784786:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076535.784797:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076535.784806:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076535.784816:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076535.784826:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076535.784836:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076535.784839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076532/real 1769076535] req@ffff995b868c5c00 x1855007616577280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076535.784907:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616577280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076535.784923:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076535.784929:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076535.784934:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076535.784943:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076532/real 1769076535] req@ffff995b868c7800 x1855007616576896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076535.784955:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616576896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076535.784964:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076535.784967:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076535.784971:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616576896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076535.784978:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076532/real 1769076535] req@ffff995b868c4a80 x1855007616577024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076535.784991:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616577024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076535.785002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076535.785006:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076535.785009:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076535.785017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076532/real 1769076535] req@ffff995b868c6d80 x1855007616577152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076535.785029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616577152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076535.785038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076535.785042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076535.785046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076535.785054:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076532/real 1769076535] req@ffff995b868c5500 x1855007616576768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076535.785075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616576768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076535.785120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076535.785124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076535.785128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616576768:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076535.785157:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076532/real 1769076535] req@ffff995b868c4700 x1855007616577536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076535.785171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616577536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076535.785180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076535.785183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076535.785187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616577536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076535.785194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076532/real 1769076535] req@ffff995b868c7480 x1855007616577408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076535.785206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616577408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076535.785214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076535.785218:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076535.785221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616577408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076536.488542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076536.488610:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076536.808548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076536.808550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076536.808550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076536.808606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076537.512532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076537.512582:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076537.832527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076537.832562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076537.832565:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076537.832572:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769076537.832573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076537.832576:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076537.832585:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3523 00000100:00080000:0.0:1769076537.832595:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076537.832601:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076537.832605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076537.832631:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616577664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076537.832652:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076537.832655:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076537.832656:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076537.832659:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3523 00000100:00080000:0.0:1769076537.832661:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076537.832664:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076537.832671:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076537.832679:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616577792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076537.832688:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616577664:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076537.832690:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076537.832697:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076537.832699:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076537.832704:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3523 00000100:00080000:0.0:1769076537.832708:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076537.832712:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076537.832717:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076537.832740:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616577920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076537.832749:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076537.832753:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076537.832755:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076537.832759:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3523 00000100:00080000:0.0:1769076537.832766:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076537.832770:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076537.832776:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076537.832779:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076537.832790:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616578048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076537.832800:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076537.832803:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076537.832805:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076537.832809:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3523 00000100:00080000:0.0:1769076537.832813:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076537.832817:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076537.832822:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076537.832827:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076537/real 1769076537] req@ffff995b84643800 x1855007616577664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076537.832836:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616578176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076537.832854:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616577664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076537.832857:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076537.832867:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076537.832871:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076537.832877:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076537.832878:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3523 00000100:00080000:0.0:1769076537.832886:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076537.832889:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076537.832894:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076537.832900:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616577664:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076537.832901:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076537.832920:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616578304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076537.832931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076537.832934:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577792:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076537.832935:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076537.832939:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076537.832945:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3523 00000100:00080000:0.0:1769076537.832950:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076537.832954:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076537.832960:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076537.832970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076537.832973:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616578432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076537.832992:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076537/real 1769076537] req@ffff995b84642a00 x1855007616577792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076537.833007:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616577792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076537.833016:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076537.833019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076537.833023:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076537.833031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076537.833070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076537.833083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076537/real 1769076537] req@ffff995b84640e00 x1855007616577920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076537.833096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616577920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076537.833107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076537.833110:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076537.833114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616577920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076537.833120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076537.833140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076537.833153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076537/real 1769076537] req@ffff995b84640000 x1855007616578048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076537.833164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616578048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076537.833172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076537.833176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076537.833179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076537.833185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578176:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076537.833379:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076537.833393:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076537/real 1769076537] req@ffff995b84641c00 x1855007616578176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076537.833406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616578176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076537.833414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076537.833419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076537.833422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578176:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076537.833442:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616578304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076537.833489:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076537.833501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076537/real 1769076537] req@ffff995b84642680 x1855007616578304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076537.833512:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616578304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076537.833521:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076537.833524:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076537.833527:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616578304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076537.833533:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616578432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076537.833551:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076537.833562:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076537/real 1769076537] req@ffff995b84640700 x1855007616578432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076537.833573:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616578432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076537.833588:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076537.833591:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076537.833595:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616578432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076538.536561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076538.536623:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076538.856545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076538.856556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076538.856557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076538.856602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076539.560521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076539.560590:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076539.880558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076539.880581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076539.880595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076539.880596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076540.584546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076540.584608:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076540.904549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076540.904549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076540.904554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076540.904584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076541.608560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076541.608655:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076541.928546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076541.928552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076541.928584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076541.928594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076542.632512:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076542.632585:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076542.952529:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076542.952541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076542.952541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076542.952555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076542.952561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076542.952567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076542.952570:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076542.952578:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3528 00000100:00080000:2.0:1769076542.952588:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076542.952594:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076542.952626:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616578560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076542.952650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076542.952655:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076542.952657:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076542.952662:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3528 00000100:00080000:2.0:1769076542.952666:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076542.952671:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076542.952681:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076542.952687:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616578560:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076542.952698:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616578688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076542.952710:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076542.952712:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076542.952714:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076542.952717:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3528 00000100:00080000:2.0:1769076542.952731:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076542.952734:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076542.952739:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076542.952743:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076542.952752:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616578816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076542.952767:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076542.952769:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076542.952771:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076542.952774:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3528 00000100:00080000:2.0:1769076542.952777:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076542.952781:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076542/real 1769076542] req@ffff995b85ac8380 x1855007616578560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076542.952782:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076542.952789:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076542.952800:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616578944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076542.952806:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616578560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076542.952812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076542.952819:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076542.952822:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076542.952828:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3528 00000100:00080000:3.0:1769076542.952829:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076542.952831:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076542.952836:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076542.952838:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:2.0:1769076542.952840:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076542.952845:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616578560:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076542.952851:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616579072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076542.952858:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076542.952860:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076542.952863:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076542.952868:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3528 00000100:00100000:3.0:1769076542.952868:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578688:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076542.952874:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076542.952894:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076542.952898:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076542.952909:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616579200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076542.952917:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076542.952922:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076542.952923:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076542.952928:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076542.952933:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3528 00000100:00080000:2.0:1769076542.952936:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076542.952939:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076542.952942:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076542.952950:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616579328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076542.952992:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076542/real 1769076542] req@ffff995b85acbb80 x1855007616578688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076542.953012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616578688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076542.953018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076542.953021:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076542.953024:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076542.953029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076542.953045:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076542.953053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076542/real 1769076542] req@ffff995b85acb100 x1855007616578816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076542.953061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616578816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076542.953068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076542.953071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076542.953073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076542.953077:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578944:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076542.953095:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076542.953102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076542/real 1769076542] req@ffff995b85ac8a80 x1855007616578944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076542.953109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616578944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076542.953119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076542.953121:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076542.953124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616578944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076542.953127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579072:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076542.953137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076542.953143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076542/real 1769076542] req@ffff995b85aca300 x1855007616579072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076542.953150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616579072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076542.953156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076542.953157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076542.953159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076542.953171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616579200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076542.953181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076542.953187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076542/real 1769076542] req@ffff995b85acb800 x1855007616579200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076542.953194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616579200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076542.953200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076542.953202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076542.953204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616579200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076542.953208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616579328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076542.953217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076542.953223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076542/real 1769076542] req@ffff995b85ac9c00 x1855007616579328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076542.953230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616579328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076542.953235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076542.953237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076542.953239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616579328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076543.656533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076543.656675:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076543.976544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076543.976591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076543.976601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076543.976637:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076544.680594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076544.680726:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076545.000562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076545.000577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076545.000593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076545.000597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076545.704573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076545.704583:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076546.024535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076546.024540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076546.024590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076546.024609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076546.728600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076546.728702:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076547.048516:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076547.048519:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076547.048562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076547.049524:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076547.752553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076547.752634:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076548.072526:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076548.072557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076548.072578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076548.072605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076548.072617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076548.072625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076548.072629:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076548.072637:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3533 00000100:00080000:3.0:1769076548.072648:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076548.072658:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076548.072693:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616579456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076548.072722:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076548.072727:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076548.072730:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076548.072734:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3533 00000100:00080000:3.0:1769076548.072739:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076548.072743:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076548.072750:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076548.072763:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616579456:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076548.072764:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616579584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076548.072781:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076548.072785:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076548.072787:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076548.072791:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3533 00000100:00080000:3.0:1769076548.072796:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076548.072800:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076548.072806:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076548.072819:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616579712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076548.072825:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076548.072837:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076548.072841:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076548.072842:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076548.072846:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3533 00000100:00080000:3.0:1769076548.072850:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076548.072853:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076548.072858:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076548.072868:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616579840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076548.072884:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076548/real 1769076548] req@ffff995b84559500 x1855007616579456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076548.072912:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076548.072916:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616579456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076548.072917:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076548.072920:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076548.072926:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3533 00000100:00080000:0.0:1769076548.072932:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076548.072932:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076548.072938:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076548.072941:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769076548.072945:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076548.072946:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616579456:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076548.072964:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616579968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076548.072972:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579584:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076548.072974:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076548.072978:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076548.072980:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076548.072984:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3533 00000100:00080000:3.0:1769076548.072987:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076548.072990:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076548.072994:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076548.072998:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076548.073008:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616580096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076548.073013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076548/real 1769076548] req@ffff995b84558a80 x1855007616579584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076548.073020:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076548.073027:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076548.073031:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076548.073037:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3533 00000100:00100000:0.0:1769076548.073038:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616579584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076548.073040:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076548.073047:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076548.073054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076548.073055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076548.073060:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076548.073064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076548.073073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076548.073073:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616580224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076548.073092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076548.073100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076548/real 1769076548] req@ffff995b8455a680 x1855007616579712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076548.073108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616579712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076548.073115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076548.073117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076548.073119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076548.073123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579840:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076548.073133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076548.073139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076548/real 1769076548] req@ffff995b8455b100 x1855007616579840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076548.073146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616579840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076548.073152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076548.073154:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076548.073156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579840:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076548.073166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579968:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076548.073176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076548.073183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076548/real 1769076548] req@ffff995b8455aa00 x1855007616579968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076548.073195:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616579968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076548.073204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076548.073207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076548.073211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616579968:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076548.073217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616580096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076548.073238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076548.073250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076548/real 1769076548] req@ffff995b8455bb80 x1855007616580096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076548.073261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616580096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076548.073267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076548.073269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076548.073271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616580096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076548.073276:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616580224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076548.073289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076548.073297:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076548/real 1769076548] req@ffff995b84558000 x1855007616580224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076548.073309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616580224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076548.073319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076548.073323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076548.073326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616580224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076548.776608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076548.776683:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076549.096510:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076549.096562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076549.096573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076549.096622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076549.800557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076549.800576:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076550.120543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076550.120559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076550.120589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076550.120623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076550.824563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076550.824600:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076551.144540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076551.144578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076551.144579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076551.144580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076551.848528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076551.848614:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076552.168515:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076552.168525:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076552.168583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076552.168635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076552.872547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076552.872614:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076553.192539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076553.192548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076553.192556:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769076553.192560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076553.192561:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076553.192575:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3538 00000100:00100000:0.0:1769076553.192576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076553.192592:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076553.192601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076553.192604:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076553.192651:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616580352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076553.192678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076553.192681:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076553.192682:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076553.192686:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3538 00000100:00080000:3.0:1769076553.192688:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076553.192694:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076553.192703:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076553.192715:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616580480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076553.192726:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616580352:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076553.192727:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076553.192739:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076553.192745:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076553.192750:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3538 00000100:00080000:3.0:1769076553.192753:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076553.192755:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076553.192758:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076553.192772:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616580608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076553.192778:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076553.192781:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076553.192782:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076553.192785:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3538 00000100:00080000:3.0:1769076553.192801:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076553.192805:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076553.192809:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076553.192817:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616580736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076553.192823:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076553.192827:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076553.192828:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076553.192835:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3538 00000800:00000100:0.0:1769076553.192836:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076553.192840:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076553.192844:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076553.192849:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076553.192858:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616580864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076553.192865:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076553.192869:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076553.192870:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076553.192876:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3538 00000100:00080000:3.0:1769076553.192878:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076553.192880:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076553.192883:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076553.192891:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616580992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076553.192898:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076553.192903:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076553.192904:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076553/real 1769076553] req@ffff995b84558700 x1855007616580352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076553.192906:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076553.192913:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3538 00000100:00080000:3.0:1769076553.192923:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076553.192931:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076553.192942:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076553.192959:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616581120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076553.192964:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616580352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076553.192986:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076553.192992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076553.192997:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616580352:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076553.193023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616580480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076553.193050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076553.193064:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076553/real 1769076553] req@ffff995b84559c00 x1855007616580480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076553.193078:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616580480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076553.193087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076553.193091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076553.193094:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616580480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076553.193102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616580608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076553.193124:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076553.193136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076553/real 1769076553] req@ffff995b8455a300 x1855007616580608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076553.193148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616580608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076553.193158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076553.193162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076553.193166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616580608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076553.193172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616580736:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076553.193194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076553.193206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076553/real 1769076553] req@ffff995b8455ad80 x1855007616580736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076553.193218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616580736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076553.193228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076553.193232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076553.193236:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616580736:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076553.193242:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616580864:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076553.193261:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076553.193273:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076553/real 1769076553] req@ffff995b84559880 x1855007616580864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076553.193285:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616580864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076553.193296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076553.193299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076553.193302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616580864:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076553.193308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616580992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076553.193327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076553.193339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076553/real 1769076553] req@ffff995b8455b480 x1855007616580992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076553.193353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616580992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076553.193361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076553.193365:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076553.193369:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616580992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076553.193375:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616581120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076553.193393:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076553.193406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076553/real 1769076553] req@ffff995b84558000 x1855007616581120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076553.193419:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616581120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076553.193428:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076553.193432:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076553.193436:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616581120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076553.896535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076553.896633:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076554.216546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076554.216554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076554.216567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076554.216598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076554.920532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076554.920623:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076555.240545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076555.240571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076555.240602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076555.240608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076555.944519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076555.944565:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076556.264550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076556.264610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076556.264613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076556.264643:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076556.968525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076556.968592:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076557.288555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076557.288570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076557.288570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076557.288586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076557.992517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076557.992569:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076558.312562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076558.312565:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076558.312567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076558.312569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076558.312577:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076558.312593:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076558.312603:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3544 00000100:00080000:2.0:1769076558.312616:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076558.312626:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076558.312630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076558.312667:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616581248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076558.312693:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076558.312698:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076558.312714:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076558.312718:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3544 00000100:00080000:2.0:1769076558.312722:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076558.312726:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076558.312734:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076558.312743:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616581248:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076558.312748:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616581376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076558.312757:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076558.312760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076558.312762:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076558.312764:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3544 00000100:00080000:2.0:1769076558.312767:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076558.312771:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076558.312776:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076558.312790:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616581504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076558.312800:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769076558.312801:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076558.312808:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076558.312811:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076558.312817:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3544 00000100:00080000:2.0:1769076558.312822:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076558.312827:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076558.312833:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076558.312839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076558/real 1769076558] req@ffff995b85acaa00 x1855007616581248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076558.312848:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616581632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076558.312857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616581248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076558.312863:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076558.312869:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076558.312870:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076558.312875:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076558.312877:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076558.312887:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616581248:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076558.312887:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3544 00000100:00080000:2.0:1769076558.312895:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076558.312902:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076558.312908:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076558.312913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616581376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076558.312921:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616581760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076558.312932:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769076558.312937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076558.312939:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076558.312943:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076558.312948:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3544 00000100:00100000:0.0:1769076558.312951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076558/real 1769076558] req@ffff995b85ac9880 x1855007616581376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076558.312955:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076558.312964:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076558.312967:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616581376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076558.312973:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076558.312976:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076558.312980:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076558.312983:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616581376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076558.312989:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616581504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076558.312992:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616581888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076558.313005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076558.313007:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076558.313012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076558.313015:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076558/real 1769076558] req@ffff995b85ac9c00 x1855007616581504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076558.313017:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076558.313028:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3544 00000100:00100000:0.0:1769076558.313033:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616581504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076558.313035:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076558.313044:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076558.313047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076558.313051:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076558.313053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076558.313058:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616581504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076558.313064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616581632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076558.313071:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616582016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076558.313080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076558.313087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076558/real 1769076558] req@ffff995b85acb800 x1855007616581632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076558.313106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616581632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076558.313113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076558.313115:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076558.313117:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616581632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076558.313130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616581760:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076558.313153:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076558.313166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076558/real 1769076558] req@ffff995b85aca300 x1855007616581760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076558.313178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616581760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076558.313187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076558.313190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076558.313194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616581760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076558.313200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616581888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076558.313220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076558.313232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076558/real 1769076558] req@ffff995b85ac8a80 x1855007616581888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076558.313244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616581888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076558.313254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076558.313256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076558.313258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616581888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076558.313262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616582016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076558.313276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076558.313283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076558/real 1769076558] req@ffff995b85acb100 x1855007616582016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076558.313300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616582016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076558.313306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076558.313308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076558.313310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616582016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076559.016520:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076559.016631:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076559.336541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076559.336549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076559.336677:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076559.336677:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076560.040579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076560.040664:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076560.360570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076560.360582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076560.360608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076560.360644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076561.064553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076561.064619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076561.384568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076561.384570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076561.384587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076561.384622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076562.088531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076562.088583:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076562.408533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076562.408533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076562.408556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076562.408580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076563.112532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076563.112606:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076563.432542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076563.432542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076563.432560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076563.432594:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076563.432594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076563.432601:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076563.432605:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076563.432613:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3549 00000100:00080000:1.0:1769076563.432636:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076563.432642:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076563.432672:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616582144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076563.432699:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076563.432702:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076563.432703:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076563.432706:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3549 00000100:00080000:1.0:1769076563.432709:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076563.432712:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076563.432720:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076563.432731:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616582144:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076563.432734:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616582272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076563.432746:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076563.432748:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076563.432750:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076563.432754:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3549 00000100:00080000:1.0:1769076563.432757:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076563.432760:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076563.432763:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076563.432771:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616582400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076563.432778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076563.432781:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076563.432782:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076563.432784:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3549 00000100:00080000:1.0:1769076563.432787:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076563.432789:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076563.432792:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076563.432802:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076563.432803:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616582528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076563.432812:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076563.432815:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076563.432816:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076563.432819:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3549 00000100:00080000:1.0:1769076563.432821:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076563.432823:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076563.432827:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076563.432840:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616582656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076563.432840:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076563/real 1769076563] req@ffff995b868c6a00 x1855007616582144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076563.432871:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076563.432874:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076563.432876:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076563.432879:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3549 00000100:00080000:1.0:1769076563.432882:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076563.432884:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616582144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076563.432888:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076563.432893:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076563.432904:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616582784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076563.432905:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076563.432912:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076563.432920:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616582144:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076563.432934:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076563.432937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076563.432938:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076563.432942:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3549 00000100:00080000:1.0:1769076563.432944:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076563.432946:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076563.432949:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076563.432954:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616582272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076563.432958:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616582912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076563.432998:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076563.433016:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076563/real 1769076563] req@ffff995b868c7100 x1855007616582272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076563.433031:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616582272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076563.433041:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076563.433046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076563.433049:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616582272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076563.433056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616582400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076563.433077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076563.433089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076563/real 1769076563] req@ffff995b868c6300 x1855007616582400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076563.433103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616582400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076563.433115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076563.433120:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076563.433125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616582400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076563.433132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616582528:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076563.433150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076563.433161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076563/real 1769076563] req@ffff995b868c7480 x1855007616582528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076563.433184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616582528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076563.433196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076563.433200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076563.433205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616582528:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076563.433212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616582656:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076563.433239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076563.433256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076563/real 1769076563] req@ffff995b868c4700 x1855007616582656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076563.433268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616582656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076563.433277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076563.433281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076563.433284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616582656:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076563.433290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616582784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076563.433308:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076563.433319:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076563/real 1769076563] req@ffff995b868c5500 x1855007616582784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076563.433332:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616582784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076563.433347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076563.433350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076563.433354:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616582784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076563.433373:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616582912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076563.433392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076563.433403:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076563/real 1769076563] req@ffff995b868c6d80 x1855007616582912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076563.433415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616582912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076563.433423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076563.433427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076563.433430:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616582912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076564.136533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076564.136582:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076564.456546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076564.456547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076564.456569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076564.456581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076565.160513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076565.160581:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076565.480543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076565.480547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076565.480549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076565.480549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076566.184537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076566.184636:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076566.504580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076566.504580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076566.504592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076566.504602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076567.208522:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076567.208590:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076567.528538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076567.528538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076567.528569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076567.528601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076568.232544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076568.232621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076568.552588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076568.552589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076568.552590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076568.552595:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076568.552607:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076568.552613:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076568.552625:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3554 00000100:00080000:0.0:1769076568.552639:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076568.552647:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076568.552694:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616583040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076568.552697:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076568.552718:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076568.552722:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076568.552723:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076568.552727:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3554 00000100:00080000:0.0:1769076568.552730:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076568.552733:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076568.552743:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076568.552744:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616583040:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076568.552755:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616583168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076568.552763:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076568.552766:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076568.552767:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076568.552769:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3554 00000100:00080000:0.0:1769076568.552772:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076568.552774:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076568.552786:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076568.552795:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616583296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076568.552803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076568.552806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076568.552807:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076568.552810:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3554 00000100:00080000:0.0:1769076568.552813:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076568.552816:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076568.552819:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076568.552828:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076568.552830:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616583424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076568.552838:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076568.552841:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076568.552842:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076568.552844:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3554 00000100:00080000:0.0:1769076568.552847:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076568.552850:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076568.552852:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076568.552859:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616583552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076568.552865:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076568.552867:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076568.552868:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076568.552872:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3554 00000100:00080000:0.0:1769076568.552875:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076568.552879:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076568.552883:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076568.552887:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076568/real 1769076568] req@ffff995b84640a80 x1855007616583040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076568.552894:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616583680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076568.552904:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076568.552907:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076568.552909:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076568.552913:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616583040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076568.552915:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3554 00000100:00080000:0.0:1769076568.552920:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076568.552926:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076568.552932:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076568.552935:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769076568.552942:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616583808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076568.552942:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076568.552952:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616583040:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076568.552981:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616583168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076568.553001:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076568.553010:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076568/real 1769076568] req@ffff995b84641f80 x1855007616583168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.553019:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616583168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.553026:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.553028:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076568.553030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616583168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076568.553034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616583296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076568.553044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076568.553050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076568/real 1769076568] req@ffff995b84640380 x1855007616583296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.553063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616583296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.553070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.553072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076568.553074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616583296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076568.553078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616583424:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076568.553088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076568.553094:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076568/real 1769076568] req@ffff995b84640700 x1855007616583424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.553102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616583424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.553109:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.553110:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076568.553112:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616583424:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076568.553116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616583552:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076568.553125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076568.553130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076568/real 1769076568] req@ffff995b84642680 x1855007616583552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.553138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616583552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.553149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.553151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076568.553153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616583552:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076568.553156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616583680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076568.553166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076568.553171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076568/real 1769076568] req@ffff995b84641c00 x1855007616583680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.553178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616583680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.553184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.553186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076568.553188:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616583680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076568.553191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616583808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076568.553216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076568.553228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076568/real 1769076568] req@ffff995b84640000 x1855007616583808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.553240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616583808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.553248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.553252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076568.553256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616583808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076569.256556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076569.256620:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076569.576591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076569.576623:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076569.576625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076569.576637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076570.280570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076570.280669:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076570.600566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076570.600578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076570.600636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076570.600637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076571.304581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076571.304650:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076571.624564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076571.624566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076571.624578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076571.624598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076572.328538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076572.328594:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076572.648571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076572.648585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076572.648626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076572.648644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076573.352591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076573.352664:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076573.672550:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076573.672551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076573.672561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076573.672565:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076573.672573:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3559 00000100:00100000:0.0:1769076573.672575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076573.672585:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076573.672595:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076573.672604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076573.672618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076573.672635:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616583936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076573.672673:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076573.672676:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076573.672678:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076573.672681:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3559 00000100:00080000:2.0:1769076573.672685:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076573.672690:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076573.672699:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076573.672702:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616583936:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076573.672713:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616584064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076573.672727:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076573.672740:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076573.672742:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076573.672745:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3559 00000100:00080000:2.0:1769076573.672748:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076573.672751:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076573.672754:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076573.672763:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616584192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076573.672769:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076573.672774:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1769076573.672775:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076573.672778:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076573.672783:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3559 00000100:00080000:2.0:1769076573.672786:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076573.672789:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076573.672793:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076573.672810:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616584320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076573.672819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076573.672822:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076573.672825:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076573.672831:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076573/real 1769076573] req@ffff995b85acbb80 x1855007616583936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076573.672832:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3559 00000100:00080000:2.0:1769076573.672840:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076573.672848:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076573.672859:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076573.672864:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616583936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076573.672874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076573.672877:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616584448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076573.672889:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076573.672897:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616583936:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076573.672905:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076573.672909:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076573.672911:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076573.672915:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3559 00000100:00080000:2.0:1769076573.672918:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076573.672921:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076573.672925:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584064:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769076573.672928:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076573.672942:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616584576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076573.672949:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769076573.672950:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076573.672956:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076573.672958:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076573.672963:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3559 00000100:00100000:0.0:1769076573.672967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076573/real 1769076573] req@ffff995b85ac8380 x1855007616584064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076573.672968:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076573.672975:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076573.672982:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076573.672988:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616584064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076573.673006:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616584704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076573.673010:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076573.673018:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076573.673025:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076573.673037:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076573.673051:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076573.673069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076573/real 1769076573] req@ffff995b85acb100 x1855007616584192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076573.673082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616584192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076573.673099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076573.673102:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076573.673106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076573.673113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584320:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076573.673134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076573.673143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076573/real 1769076573] req@ffff995b85ac8a80 x1855007616584320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076573.673152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616584320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076573.673158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076573.673161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076573.673163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584320:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076573.673167:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584448:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076573.673178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076573.673192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076573/real 1769076573] req@ffff995b85aca300 x1855007616584448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076573.673199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616584448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076573.673205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076573.673207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076573.673209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584448:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076573.673222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616584576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076573.673232:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076573.673238:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076573/real 1769076573] req@ffff995b85acb800 x1855007616584576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076573.673245:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616584576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076573.673250:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076573.673252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076573.673254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616584576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076573.673258:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616584704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076573.673267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076573.673272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076573/real 1769076573] req@ffff995b85ac9c00 x1855007616584704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076573.673280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616584704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076573.673285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076573.673286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076573.673288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616584704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076574.376648:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076574.376762:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076574.696551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076574.696554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076574.696563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076574.696568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076575.400535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076575.400579:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076575.720546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076575.720565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076575.720576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076575.720586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076576.424583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076576.424662:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076576.744569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076576.744570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076576.744586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076576.744603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076577.448537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076577.448587:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076577.768531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076577.768548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076577.768553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076577.768572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076578.472541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076578.472605:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076578.792581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076578.792583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076578.792658:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076578.792663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076578.792665:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076578.792666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076578.792676:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076578.792698:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3564 00000100:00080000:3.0:1769076578.792711:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076578.792720:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076578.792761:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616584832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076578.792806:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076578.792810:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076578.792812:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076578.792816:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3564 00000100:00080000:3.0:1769076578.792820:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076578.792824:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076578.792834:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076578.792847:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616584960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076578.792859:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076578.792863:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076578.792865:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076578.792872:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3564 00000100:00100000:0.0:1769076578.792874:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616584832:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076578.792876:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076578.792885:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076578.792892:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076578.792905:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616585088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076578.792916:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076578.792920:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076578.792922:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076578.792926:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3564 00000100:00080000:3.0:1769076578.792931:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076578.792935:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076578.792939:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076578.792949:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616585216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076578.792959:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769076578.792964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076578.792966:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076578.792971:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076578.792976:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3564 00000100:00080000:3.0:1769076578.792988:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076578.792992:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076578.792998:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076578.793011:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616585344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076578.793020:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076578.793024:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076578.793027:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076578.793033:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076578/real 1769076578] req@ffff995b8455bb80 x1855007616584832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076578.793037:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3564 00000100:00080000:3.0:1769076578.793046:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076578.793055:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076578.793063:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076578.793072:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616584832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076578.793078:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616585472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076578.793113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076578.793114:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076578.793125:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076578.793128:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076578.793133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076578.793138:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3564 00000100:00100000:0.0:1769076578.793151:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616584832:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076578.793164:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076578.793172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584960:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076578.793174:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076578.793183:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076578.793201:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616585600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076578.793237:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076578.793257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076578/real 1769076578] req@ffff995b8455aa00 x1855007616584960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076578.793272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616584960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076578.793290:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076578.793293:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076578.793297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616584960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076578.793325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076578.793391:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076578.793416:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076578/real 1769076578] req@ffff995b8455b100 x1855007616585088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076578.793436:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616585088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076578.793491:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076578.793499:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076578.793505:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076578.793513:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585216:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076578.793541:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076578.793555:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076578/real 1769076578] req@ffff995b8455a680 x1855007616585216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076578.793568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616585216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076578.793577:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076578.793580:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076578.793584:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585216:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076578.793590:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585344:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076578.793605:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076578.793614:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076578/real 1769076578] req@ffff995b84558a80 x1855007616585344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076578.793625:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616585344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076578.793635:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076578.793638:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076578.793641:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585344:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076578.793646:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616585472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076578.793660:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076578.793669:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076578/real 1769076578] req@ffff995b84559500 x1855007616585472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076578.793679:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616585472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076578.793699:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076578.793702:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076578.793705:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616585472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076578.793710:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616585600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076578.793724:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076578.793733:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076578/real 1769076578] req@ffff995b84558000 x1855007616585600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076578.793744:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616585600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076578.793752:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076578.793755:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076578.793759:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616585600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076579.496610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076579.496779:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076579.816610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076579.816610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076579.816638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076579.816687:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076580.520536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076580.520603:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076580.840534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076580.840535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076580.840541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076580.840566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076581.544544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076581.544613:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076581.864539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076581.864543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076581.864543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076581.864552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076582.568571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076582.568683:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076582.888610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076582.888654:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076582.888664:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076582.888665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076583.592598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076583.592776:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076583.912593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076583.912595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076583.912621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076583.912647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076583.912654:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076583.912662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076583.912669:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3569 00000100:00080000:3.0:1769076583.912687:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076583.912701:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076583.912702:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076583.912766:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616585728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076583.912812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076583.912820:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076583.912824:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076583.912832:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3569 00000100:00080000:3.0:1769076583.912848:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076583.912855:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076583.912870:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076583.912883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616585728:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076583.912892:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616585856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076583.912910:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076583.912915:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076583.912918:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076583.912924:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3569 00000100:00080000:3.0:1769076583.912929:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076583.912935:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076583.912942:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076583.912957:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616585984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076583.912968:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076583.912973:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076583.912978:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076583.912981:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076583.912987:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3569 00000100:00080000:3.0:1769076583.912992:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076583.912998:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076583.913005:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076583.913020:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616586112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076583.913025:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076583/real 1769076583] req@ffff995b8455b480 x1855007616585728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076583.913038:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076583.913043:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076583.913046:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076583.913052:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3569 00000100:00100000:0.0:1769076583.913054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616585728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076583.913058:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076583.913065:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076583.913072:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769076583.913073:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076583.913081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076583.913088:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616585728:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076583.913088:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616586240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076583.913103:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076583.913108:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076583.913111:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076583.913116:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3569 00000100:00100000:0.0:1769076583.913122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585856:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076583.913122:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076583.913128:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076583.913135:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076583.913152:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616586368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076583.913158:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076583.913170:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076583.913175:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076583.913179:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076583.913180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076583/real 1769076583] req@ffff995b84559880 x1855007616585856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076583.913185:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3569 00000100:00080000:3.0:1769076583.913191:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076583.913198:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076583.913203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616585856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076583.913206:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076583.913219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076583.913222:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616586496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076583.913226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076583.913233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076583.913243:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076583.913300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076583.913325:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076583/real 1769076583] req@ffff995b8455ad80 x1855007616585984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076583.913349:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616585984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076583.913369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076583.913373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076583.913377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616585984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076583.913386:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616586112:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076583.913415:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076583.913428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076583/real 1769076583] req@ffff995b8455a300 x1855007616586112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076583.913441:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616586112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076583.913482:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076583.913487:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076583.913491:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616586112:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076583.913499:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616586240:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076583.913521:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076583.913533:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076583/real 1769076583] req@ffff995b84559c00 x1855007616586240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076583.913547:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616586240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076583.913557:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076583.913561:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076583.913565:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616586240:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076583.913586:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616586368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076583.913604:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076583.913615:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076583/real 1769076583] req@ffff995b84558700 x1855007616586368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076583.913628:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616586368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076583.913637:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076583.913640:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076583.913643:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616586368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076583.913649:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616586496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076583.913665:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076583.913675:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076583/real 1769076583] req@ffff995b8455b480 x1855007616586496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076583.913696:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616586496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076583.913706:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076583.913709:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076583.913712:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616586496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076584.616575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076584.616683:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076584.936546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076584.936565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076584.936603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076584.936641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076585.640676:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076585.640783:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076585.960652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076585.960666:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076585.960673:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076585.960710:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076586.664584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076586.664724:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076586.984589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076586.984628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076586.984629:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076586.984668:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076587.688511:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076587.688601:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076588.008611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076588.008619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076588.008627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076588.008700:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076588.712517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076588.712599:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076589.032629:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076589.032669:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076589.032685:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076589.032695:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076589.032702:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076589.032709:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076589.032717:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3574 00000100:00080000:1.0:1769076589.032736:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076589.032746:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076589.032753:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076589.032807:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616586624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076589.032848:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076589.032856:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076589.032861:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076589.032882:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3574 00000100:00080000:1.0:1769076589.032892:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076589.032901:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076589.032911:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616586624:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076589.032921:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076589.032947:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616586752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076589.032964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076589.032969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076589.032972:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076589.032976:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3574 00000100:00080000:1.0:1769076589.033003:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076589.033009:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076589.033017:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076589.033022:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076589.033057:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616586880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076589.033091:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076589.033099:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076589.033103:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076589.033110:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076589/real 1769076589] req@ffff995b868c4a80 x1855007616586624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076589.033117:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3574 00000100:00080000:1.0:1769076589.033144:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076589.033154:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076589.033163:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076589.033176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616586624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076589.033217:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616587008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076589.033223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076589.033232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076589.033238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616586624:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076589.033271:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076589.033280:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1769076589.033286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616586752:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076589.033289:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076589.033305:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3574 00000100:00080000:1.0:1769076589.033317:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076589.033324:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076589.033333:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076589.033338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076589.033362:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616587136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076589.033368:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076589/real 1769076589] req@ffff995b868c7800 x1855007616586752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076589.033389:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076589.033401:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076589.033419:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076589.033434:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3574 00000100:00080000:1.0:1769076589.033442:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076589.033448:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076589.033488:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076589.033505:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616587264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076589.033511:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616586752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076589.033533:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076589.033545:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076589.033551:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076589.033562:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076589.033562:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3574 00000100:00080000:1.0:1769076589.033568:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076589.033579:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076589.033585:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769076589.033589:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076589.033597:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616586752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076589.033619:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616587392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076589.033633:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616586880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076589.033671:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076589.033698:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076589/real 1769076589] req@ffff995b868c5c00 x1855007616586880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076589.033714:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616586880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076589.033726:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076589.033739:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076589.033743:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616586880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076589.033751:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587008:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076589.033781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076589.033805:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076589/real 1769076589] req@ffff995b868c6d80 x1855007616587008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076589.033844:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616587008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076589.033896:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076589.033913:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076589.033918:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076589.033952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587136:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076589.033994:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076589.034030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076589/real 1769076589] req@ffff995b868c5500 x1855007616587136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076589.034072:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616587136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076589.034104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076589.034111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076589.034118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587136:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076589.034162:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616587264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076589.034217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076589.034234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076589/real 1769076589] req@ffff995b868c4700 x1855007616587264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076589.034250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616587264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076589.034261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076589.034265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076589.034270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616587264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076589.034277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616587392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076589.034299:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076589.034310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076589/real 1769076589] req@ffff995b868c7480 x1855007616587392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076589.034324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616587392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076589.034334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076589.034338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076589.034341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616587392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076589.736594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076589.736699:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076590.056629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076590.056693:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076590.056693:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076590.056745:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076590.760528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076590.760585:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076591.080516:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076591.080534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076591.080566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076591.080574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076591.784540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076591.784599:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076592.104560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076592.104562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076592.104579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076592.104629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076592.808565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076592.808616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076593.128539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076593.128543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076593.128560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076593.128574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076593.832527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076593.832622:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076594.152640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076594.152640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076594.152643:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076594.152644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076594.152704:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076594.152712:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076594.152716:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076594.152727:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3579 00000100:00080000:1.0:1769076594.152740:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076594.152746:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076594.152779:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616587520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076594.152807:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076594.152811:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076594.152813:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076594.152818:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3579 00000100:00080000:1.0:1769076594.152823:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076594.152828:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076594.152839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616587520:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076594.152839:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076594.152854:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616587648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076594.152863:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076594.152866:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076594.152867:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076594.152871:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3579 00000100:00080000:1.0:1769076594.152874:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076594.152877:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076594.152880:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076594.152887:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616587776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076594.152894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769076594.152899:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076594.152900:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076594.152904:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076594.152907:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3579 00000100:00080000:1.0:1769076594.152910:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076594.152913:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076594.152917:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076594.152925:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616587904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076594.152931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076594.152934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076594.152935:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076594.152938:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3579 00000100:00080000:1.0:1769076594.152942:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076594.152946:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076594.152948:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076594/real 1769076594] req@ffff995b868c6300 x1855007616587520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076594.152949:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076594.152960:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616588032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076594.152970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076594.152973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076594.152974:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076594.152980:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3579 00000100:00100000:0.0:1769076594.152982:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616587520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076594.152983:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076594.152988:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076594.152995:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076594.153004:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616588160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076594.153008:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076594.153016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076594.153017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076594.153022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076594.153025:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616587520:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076594.153026:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076594.153034:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3579 00000100:00080000:1.0:1769076594.153037:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076594.153039:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076594.153042:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076594.153048:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616588288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076594.153056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076594.153085:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076594.153114:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076594/real 1769076594] req@ffff995b868c7100 x1855007616587648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076594.153124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616587648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076594.153130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076594.153133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076594.153135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076594.153140:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076594.153155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076594.153162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076594/real 1769076594] req@ffff995b868c6a00 x1855007616587776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076594.153170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616587776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076594.153176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076594.153178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076594.153180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076594.153185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587904:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076594.153195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076594.153201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076594/real 1769076594] req@ffff995b868c7480 x1855007616587904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076594.153208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616587904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076594.153214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076594.153216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076594.153218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616587904:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076594.153221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588032:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076594.153230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076594.153236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076594/real 1769076594] req@ffff995b868c4700 x1855007616588032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076594.153243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616588032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076594.153248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076594.153250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076594.153252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588032:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076594.153255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616588160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076594.153265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076594.153270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076594/real 1769076594] req@ffff995b868c5500 x1855007616588160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076594.153278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616588160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076594.153283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076594.153285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076594.153287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616588160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076594.153300:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616588288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076594.153310:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076594.153316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076594/real 1769076594] req@ffff995b868c6d80 x1855007616588288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076594.153323:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616588288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076594.153328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076594.153330:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076594.153332:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616588288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076594.856558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076594.856652:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076595.176590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076595.176608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076595.176616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076595.176637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076595.880540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076595.880603:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076596.200578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076596.200585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076596.200600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076596.200638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076596.904595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076596.904682:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076597.224561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076597.224616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076597.224642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076597.224656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076597.928560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076597.928701:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076598.248527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076598.248529:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076598.248542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076598.248560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076598.952546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076598.952623:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076599.272585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076599.272598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076599.272611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076599.272671:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076599.272696:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076599.272700:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076599.272710:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3584 00000100:00080000:1.0:1769076599.272723:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076599.272733:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076599.272786:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616588416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076599.272795:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076599.272815:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076599.272821:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076599.272822:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076599.272827:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3584 00000100:00080000:1.0:1769076599.272832:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076599.272836:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076599.272844:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616588416:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076599.272847:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076599.272867:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616588544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076599.272880:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076599.272883:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076599.272928:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076599.272936:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3584 00000100:00080000:1.0:1769076599.272951:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076599.272959:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076599.272971:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076599.273003:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616588672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076599.273071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076599.273071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588544:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076599.273085:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076599.273090:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076599.273099:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3584 00000100:00080000:1.0:1769076599.273115:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076599.273123:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076599.273132:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076599.273146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076599.273156:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616588800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076599.273181:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076599.273185:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076599.273187:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076599.273194:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3584 00000100:00100000:3.0:1769076599.273200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588800:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769076599.273202:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076599.273232:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076599.273243:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076599.273272:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616588928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076599.273309:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076599.273317:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769076599.273318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588928:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769076599.273324:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076599.273336:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3584 00000100:00080000:1.0:1769076599.273344:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076599.273352:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076599.273360:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076599.273376:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616589056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076599.273396:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076599.273399:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076599.273401:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076599.273406:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3584 00000100:00080000:1.0:1769076599.273411:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076599.273417:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076599.273424:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076599.273431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616589056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076599.273445:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616589184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076599.275608:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616589184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076599.976576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076599.976651:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076600.296592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076600.296601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076600.296605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076600.296657:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076601.000627:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076601.000730:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076601.320575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076601.320582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076601.320628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076601.320643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076602.024571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076602.024639:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076602.344562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076602.344565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076602.344565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076602.344633:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000400:00000100:1.0:1769076602.344826:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076602.344876:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076602.344897:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076602.344951:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076602.344997:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076602.345010:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076602.345019:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076602.345029:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076602.345039:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076602.345135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076599/real 1769076602] req@ffff995b868c7480 x1855007616589184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076602.345167:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616589184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076602.345183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076602.345189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076602.345195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616589184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076602.345208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076599/real 1769076602] req@ffff995b868c4700 x1855007616589056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076602.345224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616589056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076602.345234:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076602.345239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076602.345244:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616589056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076602.345253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076599/real 1769076602] req@ffff995b868c5500 x1855007616588928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076602.345268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616588928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076602.345280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076602.345284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076602.345289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588928:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076602.345297:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076599/real 1769076602] req@ffff995b868c6d80 x1855007616588800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076602.345312:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616588800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076602.345324:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076602.345328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076602.345333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076602.345356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076599/real 1769076602] req@ffff995b868c4a80 x1855007616588672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076602.345372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616588672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076602.345385:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076602.345389:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076602.345393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076602.345402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076599/real 1769076602] req@ffff995b868c7800 x1855007616588544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076602.345418:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616588544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076602.345428:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076602.345432:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076602.345436:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616588544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076602.345444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076599/real 1769076602] req@ffff995b868c5c00 x1855007616588416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076602.345608:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616588416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076602.345618:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076602.345622:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076602.345626:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616588416:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076603.048528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076603.048583:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076603.368527:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076603.368554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076603.368594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076603.368644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076604.072574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076604.072709:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076604.392591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076604.392591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076604.392635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076604.392645:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076604.392649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076604.392660:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076604.392666:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076604.392678:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3590 00000100:00080000:2.0:1769076604.392692:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076604.392702:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076604.392745:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616589312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076604.392796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076604.392803:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076604.392805:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076604.392812:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3590 00000100:00080000:2.0:1769076604.392818:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076604.392825:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076604.392838:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076604.392858:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616589312:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076604.392873:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616589440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076604.392887:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076604.392891:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076604.392893:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076604.392898:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3590 00000100:00080000:2.0:1769076604.392923:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076604.392944:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769076604.392945:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769076604.392951:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076604.392968:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616589568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076604.392981:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076604.392985:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076604.392988:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076604.392992:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076604/real 1769076604] req@ffff995b85ac9880 x1855007616589312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076604.392994:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3590 00000100:00080000:2.0:1769076604.392998:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076604.393004:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076604.393010:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076604.393024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616589312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076604.393030:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616589696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076604.393039:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076604.393042:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076604.393045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076604.393047:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076604.393050:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076604.393051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616589312:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076604.393055:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3590 00000100:00080000:2.0:1769076604.393059:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076604.393063:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076604.393069:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076604.393078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616589440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076604.393080:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616589824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076604.393094:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076604.393098:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076604.393100:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076604.393106:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3590 00000800:00000100:3.0:1769076604.393108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076604.393111:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076604.393115:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076604.393122:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076604.393124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076604/real 1769076604] req@ffff995b85acaa00 x1855007616589440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076604.393135:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616589952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076604.393143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616589440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076604.393149:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076604.393154:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076604.393156:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076604.393156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076604.393161:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3590 00000100:00080000:3.0:1769076604.393161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076604.393166:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076604.393167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616589440:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076604.393171:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076604.393176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616589568:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769076604.393177:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076604.393189:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616590080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076604.393201:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076604.393216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076604/real 1769076604] req@ffff995b85ac9c00 x1855007616589568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076604.393234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616589568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076604.393267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076604.393272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076604.393277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616589568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076604.393286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616589696:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076604.393316:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076604.393349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076604/real 1769076604] req@ffff995b85acb800 x1855007616589696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076604.393367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616589696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076604.393381:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076604.393388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076604.393392:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616589696:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076604.393415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616589824:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076604.393445:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076604.393493:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076604/real 1769076604] req@ffff995b85aca300 x1855007616589824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076604.393532:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616589824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076604.393546:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076604.393552:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076604.393556:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616589824:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076604.393565:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616589952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076604.393595:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076604.393612:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076604/real 1769076604] req@ffff995b85ac8a80 x1855007616589952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076604.393630:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616589952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076604.393643:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076604.393647:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076604.393652:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616589952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076604.393660:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616590080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076604.393689:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076604.393707:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076604/real 1769076604] req@ffff995b85acb100 x1855007616590080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076604.393724:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616590080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076604.393743:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076604.393747:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076604.393751:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616590080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076605.096648:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076605.096713:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076605.416609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076605.416660:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076605.416701:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076605.416796:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076606.120577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076606.120638:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076606.440565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076606.440597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076606.440607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076606.440640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076607.144545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076607.144641:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076607.464553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076607.464562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076607.464565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076607.464565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076608.168561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076608.168723:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076608.488558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076608.488618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076608.488669:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076608.488677:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076609.192540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076609.192601:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076609.512597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076609.512641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076609.512666:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076609.512685:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076609.512713:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076609.512720:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076609.512725:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076609.512753:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3595 00000100:00080000:0.0:1769076609.512765:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076609.512773:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076609.512810:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616590208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076609.512837:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076609.512859:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076609.512862:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076609.512867:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3595 00000100:00080000:0.0:1769076609.512872:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076609.512877:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076609.512887:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076609.512889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616590208:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076609.512904:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616590336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076609.512923:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076609.512926:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076609.512928:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076609.512932:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3595 00000100:00080000:0.0:1769076609.512954:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076609.512957:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076609.512963:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076609.512977:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616590464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076609.512984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076609.512997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076609.513002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076609.513005:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076609.513010:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3595 00000100:00080000:0.0:1769076609.513014:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076609.513017:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076609.513023:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076609.513045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076609/real 1769076609] req@ffff995b84640e00 x1855007616590208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076609.513046:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616590592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076609.513090:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076609.513095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076609.513097:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076609.513103:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3595 00000100:00100000:3.0:1769076609.513127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616590208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076609.513131:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076609.513140:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076609.513148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769076609.513149:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769076609.513159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076609.513165:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616590720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076609.513167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616590208:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076609.513185:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076609.513203:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076609.513206:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076609.513211:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3595 00000100:00080000:0.0:1769076609.513214:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076609.513219:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076609.513224:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076609.513235:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616590336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076609.513237:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616590848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076609.513255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076609.513258:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076609.513261:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076609.513266:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3595 00000100:00080000:0.0:1769076609.513272:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076609.513278:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076609.513279:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076609.513289:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076609.513320:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616590976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076609.513327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076609/real 1769076609] req@ffff995b84642a00 x1855007616590336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076609.513373:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616590336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076609.513409:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076609.513413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076609.513418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616590336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076609.513424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616590464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076609.513446:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076609.513456:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076609/real 1769076609] req@ffff995b84643800 x1855007616590464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076609.513493:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616590464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076609.513505:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076609.513509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076609.513513:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616590464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076609.513520:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616590592:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076609.513552:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076609.513570:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076609/real 1769076609] req@ffff995b84641180 x1855007616590592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076609.513591:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616590592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076609.513622:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076609.513627:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076609.513633:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616590592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076609.513641:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616590720:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076609.513664:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076609.513680:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076609/real 1769076609] req@ffff995b84641500 x1855007616590720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076609.513697:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616590720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076609.513710:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076609.513714:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076609.513718:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616590720:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076609.513761:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616590848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076609.513789:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076609.513801:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076609/real 1769076609] req@ffff995b84643b80 x1855007616590848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076609.513814:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616590848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076609.513823:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076609.513826:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076609.513829:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616590848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076609.513835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616590976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076609.513894:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076609.513906:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076609/real 1769076609] req@ffff995b84641880 x1855007616590976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076609.513916:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616590976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076609.513924:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076609.513927:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076609.513930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616590976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076610.216526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076610.216596:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076610.536557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076610.536562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076610.536574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076610.536608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076611.240584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076611.240687:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076611.560586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076611.560591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076611.560597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076611.560619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076612.264575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076612.264581:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076612.584562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076612.584576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076612.584597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076612.584647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076613.288529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076613.288625:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076613.608552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076613.608558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076613.608588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076613.608719:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076614.312536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076614.312584:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076614.632517:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076614.632517:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076614.632519:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076614.632531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076614.632537:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076614.632541:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076614.632550:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3600 00000100:00080000:0.0:1769076614.632562:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076614.632563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076614.632573:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076614.632623:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616591104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076614.632642:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076614.632646:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076614.632647:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076614.632650:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3600 00000100:00080000:0.0:1769076614.632653:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076614.632657:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076614.632664:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076614.632669:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616591104:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076614.632676:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616591232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076614.632684:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076614.632686:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076614.632688:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076614.632691:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3600 00000100:00080000:0.0:1769076614.632694:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076614.632696:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076614.632700:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076614.632708:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616591360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076614.632721:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076614.632724:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076614.632725:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076614.632729:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3600 00000100:00080000:0.0:1769076614.632734:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076614.632737:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076614.632741:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076614.632742:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076614.632752:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616591488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076614.632774:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076614.632776:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076614.632777:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076614.632780:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3600 00000100:00080000:0.0:1769076614.632784:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076614.632787:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076614.632791:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076614.632795:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076614/real 1769076614] req@ffff995b84642d80 x1855007616591104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076614.632803:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616591616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076614.632811:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076614.632815:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769076614.632815:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616591104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076614.632818:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076614.632826:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3600 00000100:00080000:0.0:1769076614.632831:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076614.632836:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076614.632836:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076614.632845:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076614.632849:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616591104:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076614.632850:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076614.632861:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616591744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076614.632867:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076614.632870:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076614.632872:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076614.632873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616591232:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076614.632878:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3600 00000100:00080000:0.0:1769076614.632883:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076614.632886:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076614.632890:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076614.632899:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616591872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076614.632918:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076614.632936:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076614/real 1769076614] req@ffff995b84643100 x1855007616591232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076614.632951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616591232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076614.632961:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076614.632965:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076614.632969:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616591232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076614.632976:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616591360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076614.632994:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076614.633002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076614/real 1769076614] req@ffff995b84643480 x1855007616591360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076614.633010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616591360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076614.633018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076614.633021:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076614.633023:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616591360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076614.633027:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616591488:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076614.633038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076614.633044:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076614/real 1769076614] req@ffff995b84641880 x1855007616591488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076614.633056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616591488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076614.633062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076614.633064:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076614.633066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616591488:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076614.633070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616591616:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076614.633079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076614.633085:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076614/real 1769076614] req@ffff995b84643b80 x1855007616591616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076614.633092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616591616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076614.633098:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076614.633099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076614.633101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616591616:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076614.633105:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616591744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076614.633114:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076614.633120:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076614/real 1769076614] req@ffff995b84641500 x1855007616591744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076614.633127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616591744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076614.633137:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076614.633139:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076614.633141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616591744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076614.633155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616591872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076614.633166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076614.633172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076614/real 1769076614] req@ffff995b84641180 x1855007616591872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076614.633178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616591872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076614.633184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076614.633186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076614.633188:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616591872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076615.336555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076615.336634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076615.656544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076615.656559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076615.656598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076615.656651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076616.360506:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076616.360608:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076616.680550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076616.680553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076616.680556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076616.680564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076617.384529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076617.384562:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076617.704541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076617.704547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076617.704557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076617.704568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076618.408563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076618.408742:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076618.728676:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076618.728730:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076618.728746:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076618.728785:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076619.432603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076619.432746:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076619.752574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076619.752645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076619.752657:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076619.752742:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076619.752751:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076619.752752:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076619.752764:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076619.752786:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3605 00000100:00080000:2.0:1769076619.752805:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076619.752817:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076619.752873:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616592000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076619.752918:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076619.752923:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076619.752925:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076619.752930:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3605 00000100:00080000:2.0:1769076619.752934:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076619.752939:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076619.752951:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076619.752968:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616592000:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076619.752969:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616592128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076619.752996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076619.752999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076619.753001:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076619.753005:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3605 00000100:00080000:2.0:1769076619.753008:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076619.753011:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076619.753027:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076619.753041:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616592256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076619.753051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076619.753055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076619.753056:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076619.753060:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3605 00000100:00080000:2.0:1769076619.753071:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076619.753076:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076619.753081:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076619.753095:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076619.753096:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616592384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076619.753109:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076619.753113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076619.753115:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076619.753120:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3605 00000100:00080000:2.0:1769076619.753124:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076619.753129:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076619.753134:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076619.753145:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616592512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076619.753154:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076619.753158:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076619.753160:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076619.753163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076619/real 1769076619] req@ffff995b85ac8380 x1855007616592000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076619.753168:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3605 00000100:00080000:2.0:1769076619.753175:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076619.753184:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076619.753194:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076619.753205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616592000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076619.753210:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616592640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076619.753251:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076619.753253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076619.753257:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076619.753262:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076619.753264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076619.753271:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3605 00000100:00080000:2.0:1769076619.753278:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076619.753279:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616592000:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076619.753289:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076619.753295:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076619.753315:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616592768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076619.753330:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616592128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076619.753365:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076619.753381:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076619/real 1769076619] req@ffff995b85acbb80 x1855007616592128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076619.753394:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616592128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076619.753402:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076619.753406:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076619.753409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616592128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076619.753415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616592256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076619.753430:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076619.753438:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076619/real 1769076619] req@ffff995b85acb100 x1855007616592256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076619.753449:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616592256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076619.753490:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076619.753494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076619.753497:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616592256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076619.753514:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616592384:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076619.753533:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076619.753543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076619/real 1769076619] req@ffff995b85ac8a80 x1855007616592384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076619.753555:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616592384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076619.753564:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076619.753567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076619.753570:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616592384:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076619.753576:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616592512:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076619.753590:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076619.753598:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076619/real 1769076619] req@ffff995b85aca300 x1855007616592512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076619.753608:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616592512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076619.753616:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076619.753619:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076619.753622:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616592512:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076619.753627:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616592640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076619.753640:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076619.753649:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076619/real 1769076619] req@ffff995b85acb800 x1855007616592640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076619.753661:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616592640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076619.753691:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076619.753694:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076619.753697:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616592640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076619.753717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616592768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076619.753735:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076619.753746:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076619/real 1769076619] req@ffff995b85ac9c00 x1855007616592768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076619.753757:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616592768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076619.753765:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076619.753768:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076619.753771:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616592768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076620.456643:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076620.456796:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076620.776594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076620.776605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076620.776633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076620.776716:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076621.480567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076621.480660:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076621.800596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076621.800596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076621.800615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076621.800716:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076622.504574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076622.504621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076622.824578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076622.824582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076622.824685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076622.824686:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076623.528573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076623.528678:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076623.848594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076623.848596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076623.848644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076623.848683:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076624.552568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076624.552718:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076624.872571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076624.872675:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076624.872694:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076624.872706:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076624.872713:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076624.872726:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3610 00000100:00100000:1.0:1769076624.872764:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076624.872768:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076624.872776:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076624.872789:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076624.872848:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616592896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076624.872893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076624.872901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076624.872905:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076624.872914:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3610 00000100:00080000:2.0:1769076624.872921:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076624.872929:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076624.872945:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076624.872957:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616592896:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076624.872968:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616593024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076624.872988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076624.872995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076624.872998:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076624.873005:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3610 00000100:00080000:2.0:1769076624.873011:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076624.873017:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076624.873025:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076624.873057:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616593152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076624.873075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076624.873078:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076624.873082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076624.873085:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076624.873089:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3610 00000100:00080000:2.0:1769076624.873094:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076624.873099:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076624.873105:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076624.873120:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616593280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076624.873146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076624/real 1769076624] req@ffff995b85acaa00 x1855007616592896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076624.873153:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076624.873160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076624.873163:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076624.873169:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3610 00000100:00080000:2.0:1769076624.873175:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076624.873177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616592896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076624.873182:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076624.873192:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769076624.873197:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076624.873205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076624.873207:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616593408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076624.873213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616592896:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076624.873225:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076624.873230:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076624.873233:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076624.873240:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3610 00000100:00080000:2.0:1769076624.873247:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076624.873249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593024:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076624.873254:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076624.873262:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076624.873279:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616593536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076624.873305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076624.873314:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076624.873319:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076624.873322:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076624.873324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076624/real 1769076624] req@ffff995b85ac9880 x1855007616593024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076624.873327:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3610 00000100:00080000:2.0:1769076624.873332:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076624.873338:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076624.873344:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616593024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076624.873345:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076624.873359:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616593664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076624.873359:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076624.873365:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076624.873387:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076624.873415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076624.873457:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076624.873518:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076624/real 1769076624] req@ffff995b85ac9c00 x1855007616593152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076624.873536:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616593152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076624.873557:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076624.873561:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076624.873566:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076624.873574:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593280:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076624.873666:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076624.873695:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076624/real 1769076624] req@ffff995b85acb800 x1855007616593280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076624.873715:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616593280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076624.873730:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076624.873733:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076624.873737:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076624.873771:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593408:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076624.873798:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076624.873812:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076624/real 1769076624] req@ffff995b85aca300 x1855007616593408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076624.873827:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616593408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076624.873838:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076624.873842:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076624.873847:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076624.873869:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616593536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076624.873889:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076624.873901:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076624/real 1769076624] req@ffff995b85ac8a80 x1855007616593536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076624.873915:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616593536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076624.873925:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076624.873928:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076624.873932:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616593536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076624.873938:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616593664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076624.873955:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076624.873965:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076624/real 1769076624] req@ffff995b85acb100 x1855007616593664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076624.873979:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616593664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076624.873989:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076624.873992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076624.873996:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616593664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076625.576668:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076625.576786:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076625.896649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076625.896673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076625.896717:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076625.896732:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076626.600529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076626.600603:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076626.920571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076626.920574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076626.920605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076626.920634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076627.624525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076627.624583:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076627.944632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076627.944633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076627.944668:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076627.944736:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076628.648632:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076628.648750:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076628.968597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076628.968621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076628.968624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076628.968698:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076629.672601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076629.672660:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076629.992681:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076629.992682:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076629.992683:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076629.992731:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076629.992740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076629.992745:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076629.992762:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076629.992763:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3615 00000100:00080000:2.0:1769076629.992805:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076629.992815:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076629.992890:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616593792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076629.992949:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076629.992955:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076629.992959:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076629.992966:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3615 00000100:00080000:2.0:1769076629.992973:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076629.992981:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076629.992995:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076629.992997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616593792:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076629.993020:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616593920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076629.993034:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076629.993038:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076629.993040:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076629.993044:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3615 00000100:00080000:2.0:1769076629.993061:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076629.993066:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076629.993071:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076629.993074:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076629.993092:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616594048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076629.993104:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076629.993108:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076629.993109:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076629.993114:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3615 00000100:00080000:2.0:1769076629.993118:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076629.993126:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076629.993127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076629/real 1769076629] req@ffff995b85acbb80 x1855007616593792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076629.993132:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076629.993152:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616594176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076629.993160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616593792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076629.993174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076629.993182:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076629.993183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076629.993193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769076629.993194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616593792:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076629.993198:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076629.993208:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3615 00000100:00080000:2.0:1769076629.993213:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076629.993216:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076629.993221:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076629.993228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076629.993237:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616594304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076629.993257:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076629.993262:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076629.993264:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076629.993269:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076629.993276:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3615 00000100:00080000:2.0:1769076629.993281:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076629.993284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076629/real 1769076629] req@ffff995b85ac8380 x1855007616593920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076629.993289:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076629.993298:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076629.993317:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616594432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076629.993325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616593920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076629.993332:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076629.993343:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076629.993348:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076629.993359:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3615 00000100:00080000:2.0:1769076629.993363:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076629.993363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076629.993371:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076629.993377:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076629.993379:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076629.993415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616593920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076629.993419:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616594560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076629.993429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076629.993461:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076629.993532:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076629/real 1769076629] req@ffff995b85acb100 x1855007616594048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076629.993552:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616594048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076629.993567:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076629.993572:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076629.993577:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076629.993586:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594176:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076629.993610:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076629.993621:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076629/real 1769076629] req@ffff995b85ac8a80 x1855007616594176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076629.993632:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616594176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076629.993640:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076629.993643:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076629.993646:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594176:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076629.993661:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594304:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076629.993674:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076629.993683:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076629/real 1769076629] req@ffff995b85aca300 x1855007616594304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076629.993700:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616594304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076629.993708:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076629.993710:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076629.993713:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594304:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076629.993717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616594432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076629.993730:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076629.993738:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076629/real 1769076629] req@ffff995b85acb800 x1855007616594432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076629.993748:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616594432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076629.993756:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076629.993758:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076629.993761:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616594432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076629.993765:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616594560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076629.993778:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076629.993806:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076629/real 1769076629] req@ffff995b85ac9c00 x1855007616594560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076629.993819:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616594560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076629.993836:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076629.993840:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076629.993844:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616594560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076630.696552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076630.696619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076631.016602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076631.016613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076631.016621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076631.016661:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076631.720548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076631.720596:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076632.040553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076632.040553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076632.040655:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076632.040699:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076632.744581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076632.744658:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076633.064539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076633.064542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076633.064567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076633.064586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076633.768621:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076633.768648:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076634.088579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076634.088619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076634.088631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076634.088654:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076634.792579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076634.792658:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076635.112603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076635.112641:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076635.112645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076635.112650:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076635.112657:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076635.112668:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3620 00000100:00100000:1.0:1769076635.112679:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076635.112681:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076635.112691:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076635.112715:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076635.112743:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616594688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076635.112786:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076635.112790:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076635.112792:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076635.112797:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3620 00000100:00080000:2.0:1769076635.112803:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076635.112808:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076635.112819:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076635.112846:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616594688:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076635.112850:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616594816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076635.112868:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076635.112872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076635.112874:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076635.112881:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3620 00000100:00080000:2.0:1769076635.112887:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076635.112892:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076635.112898:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076635.112924:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616594944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076635.112935:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076635.112936:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076635.112942:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076635.112945:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076635.112951:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3620 00000100:00080000:2.0:1769076635.112958:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076635.112964:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076635.112970:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076635.112985:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616595072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076635.112990:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076635/real 1769076635] req@ffff995b85ac9880 x1855007616594688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076635.113010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076635.113015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076635.113017:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076635.113024:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3620 00000100:00100000:1.0:1769076635.113026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616594688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076635.113030:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076635.113042:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769076635.113046:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076635.113048:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769076635.113055:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076635.113063:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616595200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076635.113064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616594688:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076635.113088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076635.113091:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076635.113093:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076635.113100:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3620 00000100:00100000:1.0:1769076635.113101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594816:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076635.113104:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076635.113110:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076635.113115:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076635.113131:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616595328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076635.113138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076635.113153:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076635.113157:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076635.113159:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076635.113165:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076635/real 1769076635] req@ffff995b85acaa00 x1855007616594816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076635.113168:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3620 00000100:00080000:2.0:1769076635.113177:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076635.113185:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076635.113191:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076635.113192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616594816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076635.113207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076635.113210:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616595456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076635.113215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076635.113228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076635.113239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076635.113262:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076635.113281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076635/real 1769076635] req@ffff995b85ac9c00 x1855007616594944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076635.113297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616594944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076635.113315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076635.113319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076635.113325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616594944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076635.113332:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595072:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076635.113347:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076635.113356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076635/real 1769076635] req@ffff995b85acb800 x1855007616595072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076635.113366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616595072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076635.113374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076635.113377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076635.113380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076635.113385:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595200:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076635.113397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076635.113405:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076635/real 1769076635] req@ffff995b85aca300 x1855007616595200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076635.113451:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616595200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076635.113485:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076635.113489:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076635.113495:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595200:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076635.113521:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616595328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076635.113539:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076635.113550:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076635/real 1769076635] req@ffff995b85ac8a80 x1855007616595328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076635.113568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616595328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076635.113576:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076635.113579:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076635.113582:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616595328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076635.113587:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616595456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076635.113599:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076635.113606:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076635/real 1769076635] req@ffff995b85acb100 x1855007616595456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076635.113616:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616595456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076635.113629:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076635.113632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076635.113635:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616595456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076635.816545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076635.816672:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076636.136594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076636.136634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076636.136681:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076636.136711:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076636.840567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076636.840675:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076637.160538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076637.160553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076637.160557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076637.160574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769076637.864667:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076637.864687:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076638.184580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076638.184630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076638.184632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076638.184781:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076638.888548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076638.888611:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076639.208557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076639.208560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076639.208584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076639.208593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076639.912588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076639.912703:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076640.232538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076640.232542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076640.232563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076640.232564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076640.232582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076640.232587:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076640.232607:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076640.232615:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3625 00000100:00080000:1.0:1769076640.232625:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076640.232632:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076640.232690:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616595584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076640.232711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076640.232714:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076640.232715:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076640.232718:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3625 00000100:00080000:1.0:1769076640.232722:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076640.232725:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076640.232734:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076640.232744:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616595712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076640.232754:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076640.232755:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616595584:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076640.232760:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076640.232764:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076640.232768:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3625 00000100:00080000:1.0:1769076640.232770:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076640.232772:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076640.232775:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076640.232785:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616595840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076640.232792:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076640.232795:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076640.232796:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076640.232799:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3625 00000100:00080000:1.0:1769076640.232802:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076640.232804:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076640.232807:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076640.232817:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616595968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076640.232823:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076640.232834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076640.232836:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076640.232838:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076640.232842:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3625 00000100:00080000:1.0:1769076640.232844:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076640.232846:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076640.232849:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076640.232858:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616596096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076640.232864:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076640/real 1769076640] req@ffff995b868c6a00 x1855007616595584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076640.232868:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076640.232875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076640.232878:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076640.232886:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3625 00000100:00080000:1.0:1769076640.232889:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076640.232894:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076640.232895:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616595584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076640.232901:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076640.232911:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076640.232915:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616596224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076640.232918:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076640.232925:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616595584:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076640.232938:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076640.232941:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076640.232944:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076640.232949:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595712:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076640.232951:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3625 00000100:00080000:1.0:1769076640.232959:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076640.232964:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076640.232971:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076640.232974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076640.233003:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616596352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076640.233004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076640/real 1769076640] req@ffff995b868c7100 x1855007616595712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076640.233039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616595712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076640.233062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076640.233065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076640.233068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076640.233074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076640.233086:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076640.233093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076640/real 1769076640] req@ffff995b868c6300 x1855007616595840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076640.233100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616595840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076640.233107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076640.233109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076640.233111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076640.233130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595968:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076640.233141:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076640.233147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076640/real 1769076640] req@ffff995b868c5c00 x1855007616595968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076640.233159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616595968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076640.233165:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076640.233167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076640.233169:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616595968:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076640.233172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596096:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076640.233191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076640.233197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076640/real 1769076640] req@ffff995b868c7800 x1855007616596096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076640.233204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616596096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076640.233210:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076640.233211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076640.233213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596096:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076640.233216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616596224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076640.233225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076640.233231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076640/real 1769076640] req@ffff995b868c4a80 x1855007616596224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076640.233254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616596224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076640.233276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076640.233278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076640.233281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616596224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076640.233293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616596352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076640.233305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076640.233312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076640/real 1769076640] req@ffff995b868c6d80 x1855007616596352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076640.233320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616596352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076640.233506:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076640.233509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076640.233512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616596352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076640.936602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076640.936801:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076641.256556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076641.256569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076641.256575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076641.256586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076641.960611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076641.960837:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076642.280539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076642.280634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076642.280689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076642.280741:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769076642.984573:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076642.984603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076643.304516:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076643.304533:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076643.304539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076643.304560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076644.008584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076644.008700:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076644.328577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076644.328624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076644.328659:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076644.328770:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076645.032611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076645.032735:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076645.352554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076645.352554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076645.352562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076645.352569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076645.352587:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076645.352592:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076645.352596:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076645.352604:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3631 00000100:00080000:1.0:1769076645.352615:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076645.352622:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076645.352653:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616596480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076645.352675:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076645.352678:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076645.352680:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076645.352683:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3631 00000100:00080000:1.0:1769076645.352685:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076645.352690:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076645.352698:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076645.352705:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616596480:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076645.352709:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616596608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076645.352719:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076645.352721:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076645.352722:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076645.352725:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3631 00000100:00080000:1.0:1769076645.352727:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076645.352729:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076645.352732:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076645.352741:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616596736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076645.352747:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076645.352750:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076645.352751:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076645.352753:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3631 00000100:00080000:1.0:1769076645.352757:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076645.352759:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076645.352762:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076645.352767:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076645.352773:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616596864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076645.352781:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076645.352784:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076645.352786:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076645.352788:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3631 00000100:00080000:1.0:1769076645.352791:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076645.352793:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076645.352798:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076645.352805:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076645/real 1769076645] req@ffff995b868c5500 x1855007616596480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076645.352809:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616596992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076645.352829:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076645.352836:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076645.352840:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076645.352844:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3631 00000100:00080000:1.0:1769076645.352847:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076645.352852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616596480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076645.352853:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076645.352864:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076645.352865:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076645.352873:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076645.352876:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616596480:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076645.352884:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616597120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076645.352897:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076645.352901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076645.352903:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076645.352906:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3631 00000100:00080000:1.0:1769076645.352910:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076645.352914:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076645.352919:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076645.352926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076645.352932:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616597248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076645.352951:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076645.352963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076645/real 1769076645] req@ffff995b868c4700 x1855007616596608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076645.352972:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616596608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076645.352979:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076645.352981:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076645.352983:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076645.352988:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076645.352999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076645.353005:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076645/real 1769076645] req@ffff995b868c7480 x1855007616596736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076645.353012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616596736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076645.353023:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076645.353026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076645.353028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076645.353031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596864:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076645.353041:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076645.353047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076645/real 1769076645] req@ffff995b868c6d80 x1855007616596864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076645.353054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616596864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076645.353060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076645.353062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076645.353064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596864:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076645.353069:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596992:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076645.353088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076645.353100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076645/real 1769076645] req@ffff995b868c4a80 x1855007616596992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076645.353113:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616596992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076645.353123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076645.353127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076645.353131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616596992:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076645.353148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616597120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076645.353164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076645.353172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076645/real 1769076645] req@ffff995b868c7800 x1855007616597120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076645.353179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616597120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076645.353185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076645.353187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076645.353189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616597120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076645.353193:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616597248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076645.353202:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076645.353208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076645/real 1769076645] req@ffff995b868c5c00 x1855007616597248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076645.353215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616597248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076645.353220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076645.353222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076645.353224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616597248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076646.056547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076646.056620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076646.376530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076646.376534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076646.376541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076646.376578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076647.080562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076647.080659:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076647.400543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076647.400570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076647.400571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076647.400582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076648.104540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076648.104620:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076648.424526:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076648.424528:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076648.424547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076648.424561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076649.128557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076649.128653:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076649.448536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076649.448553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076649.448579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076649.448820:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076650.152517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076650.152573:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076650.472587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076650.472604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076650.472613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076650.472622:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076650.472628:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076650.472630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076650.472640:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3636 00000100:00080000:1.0:1769076650.472655:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076650.472666:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076650.472688:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076650.472705:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616597376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076650.472731:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076650.472734:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076650.472736:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076650.472739:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3636 00000100:00080000:1.0:1769076650.472741:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076650.472745:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076650.472754:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076650.472765:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616597504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076650.472774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076650.472778:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616597376:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076650.472779:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076650.472785:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076650.472791:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3636 00000100:00080000:1.0:1769076650.472793:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076650.472795:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076650.472798:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076650.472807:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616597632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076650.472816:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076650.472819:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076650.472820:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076650.472822:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3636 00000100:00080000:1.0:1769076650.472824:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076650.472826:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076650.472829:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076650.472836:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616597760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076650.472843:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076650.472845:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076650.472846:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076650.472850:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3636 00000100:00080000:1.0:1769076650.472856:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076650.472858:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076650.472862:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076650.472867:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616597888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076650.472874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076650.472876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076650.472877:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076650.472880:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3636 00000100:00080000:1.0:1769076650.472883:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076650.472885:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076650.472890:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076650.472892:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076650.472908:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616598016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076650.472919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076650.472922:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076650.472925:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076650.472929:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3636 00000100:00080000:1.0:1769076650.472932:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076650.472934:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076650.472939:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076650.472951:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616598144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076650.472960:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076650/real 1769076650] req@ffff995b868c6300 x1855007616597376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076650.472983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616597376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076650.472993:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076650.472998:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076650.473002:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616597376:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076650.473029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616597504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076650.473050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076650.473061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076650/real 1769076650] req@ffff995b868c7100 x1855007616597504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076650.473071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616597504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076650.473077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076650.473080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076650.473082:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616597504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076650.473111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616597632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076650.473131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076650.473143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076650/real 1769076650] req@ffff995b868c6a00 x1855007616597632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076650.473172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616597632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076650.473185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076650.473188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076650.473191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616597632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076650.473197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616597760:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076650.473214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076650.473223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076650/real 1769076650] req@ffff995b868c5c00 x1855007616597760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076650.473232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616597760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076650.473240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076650.473243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076650.473246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616597760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076650.473250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616597888:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076650.473266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076650.473278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076650/real 1769076650] req@ffff995b868c7800 x1855007616597888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076650.473293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616597888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076650.473301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076650.473303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076650.473306:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616597888:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076650.473310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616598016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076650.473323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076650.473331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076650/real 1769076650] req@ffff995b868c4a80 x1855007616598016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076650.473341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616598016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076650.473347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076650.473350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076650.473352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616598016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076650.473357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616598144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076650.473368:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076650.473375:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076650/real 1769076650] req@ffff995b868c6d80 x1855007616598144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076650.473384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616598144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076650.473395:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076650.473398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076650.473401:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616598144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076651.176519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076651.176623:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076651.496539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076651.496552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076651.496565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076651.496573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076652.200524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076652.200572:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076652.520544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076652.520546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076652.520548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076652.520625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076653.224525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076653.224582:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076653.544540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076653.544540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076653.544551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076653.544579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076654.248550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076654.248591:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076654.568591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076654.568595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076654.568619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076654.568662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076655.272566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076655.272622:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076655.592528:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076655.592538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076655.592562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076655.592572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076655.592575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076655.592581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076655.592585:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076655.592593:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3641 00000100:00080000:2.0:1769076655.592602:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076655.592608:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076655.592639:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616598272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076655.592660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076655.592663:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076655.592664:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076655.592667:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3641 00000100:00080000:2.0:1769076655.592670:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076655.592673:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076655.592681:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076655.592690:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616598400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076655.592698:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616598272:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076655.592701:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076655.592708:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076655.592709:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076655.592712:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3641 00000100:00080000:2.0:1769076655.592715:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076655.592717:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076655.592720:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076655.592729:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616598528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076655.592737:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076655.592739:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076655.592741:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076655.592743:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3641 00000100:00080000:2.0:1769076655.592762:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076655.592764:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076655.592768:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076655.592775:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616598656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076655.592782:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076655.592786:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076655.592790:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076655.592791:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076655.592802:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3641 00000100:00080000:2.0:1769076655.592805:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076655.592808:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076655.592811:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076655.592820:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616598784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076655.592827:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076655.592831:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769076655.592832:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076655/real 1769076655] req@ffff995b85ac8380 x1855007616598272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076655.592833:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076655.592841:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3641 00000100:00080000:2.0:1769076655.592847:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076655.592853:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076655.592857:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076655.592864:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616598272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076655.592867:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616598912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076655.592891:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076655.592894:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076655.592900:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076655.592900:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076655.592905:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076655.592910:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616598272:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076655.592912:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3641 00000100:00080000:2.0:1769076655.592920:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076655.592923:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076655.592926:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076655.592933:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616599040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076655.592941:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616598400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076655.592968:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076655.592981:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076655/real 1769076655] req@ffff995b85acbb80 x1855007616598400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076655.592994:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616598400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076655.593003:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076655.593007:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076655.593011:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616598400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076655.593017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616598528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076655.593032:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076655.593041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076655/real 1769076655] req@ffff995b85acb100 x1855007616598528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076655.593053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616598528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076655.593064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076655.593084:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076655.593088:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616598528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076655.593095:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616598656:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076655.593114:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076655.593125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076655/real 1769076655] req@ffff995b85ac8a80 x1855007616598656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076655.593138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616598656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076655.593147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076655.593150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076655.593154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616598656:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076655.593161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616598784:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076655.593175:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076655.593185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076655/real 1769076655] req@ffff995b85aca300 x1855007616598784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076655.593197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616598784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076655.593207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076655.593210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076655.593213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616598784:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076655.593219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616598912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076655.593234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076655.593244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076655/real 1769076655] req@ffff995b85acb800 x1855007616598912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076655.593255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616598912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076655.593264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076655.593268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076655.593271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616598912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076655.593289:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616599040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076655.593305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076655.593315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076655/real 1769076655] req@ffff995b85ac9c00 x1855007616599040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076655.593327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616599040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076655.593336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076655.593338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076655.593342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616599040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076656.296533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076656.296589:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076656.616565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076656.616570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076656.616572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076656.616628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076657.320551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076657.320617:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076657.640535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076657.640543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076657.640558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076657.640591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076658.344554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076658.344644:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076658.664556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076658.664556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076658.664571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076658.664581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076659.368543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076659.368613:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076659.688566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076659.688566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076659.688566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076659.688607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076660.392524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076660.392584:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076660.712535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076660.712538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076660.712545:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076660.712545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076660.712554:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076660.712559:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076660.712577:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3646 00000100:00100000:0.0:1769076660.712582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076660.712589:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076660.712598:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076660.712634:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616599168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076660.712656:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076660.712661:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076660.712663:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076660.712667:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3646 00000100:00080000:1.0:1769076660.712671:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076660.712675:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076660.712684:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076660.712698:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616599296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076660.712710:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076660.712713:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076660.712715:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076660.712719:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3646 00000100:00080000:1.0:1769076660.712723:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076660.712727:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076660.712732:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076660.712743:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616599424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076660.712748:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616599168:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076660.712754:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076660.712758:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076660.712769:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076660.712773:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3646 00000100:00080000:1.0:1769076660.712776:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076660.712779:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076660.712784:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076660.712795:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616599552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076660.712802:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076660.712805:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076660.712807:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769076660.712809:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076660.712811:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3646 00000100:00080000:1.0:1769076660.712813:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076660.712817:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076660.712821:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076660.712830:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616599680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076660.712838:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076660.712841:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076660.712843:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076660.712846:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3646 00000100:00080000:1.0:1769076660.712849:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076660.712853:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076660.712856:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076660/real 1769076660] req@ffff995b868c7480 x1855007616599168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076660.712857:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076660.712866:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616599808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076660.712873:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616599168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076660.712874:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076660.712878:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076660.712880:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076660.712883:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076660.712884:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3646 00000100:00080000:1.0:1769076660.712887:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076660.712888:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076660.712891:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076660.712892:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616599168:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076660.712895:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076660.712898:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616599296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076660.712903:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616599936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076660.712920:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076660.712934:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076660/real 1769076660] req@ffff995b868c4700 x1855007616599296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076660.712948:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616599296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076660.712956:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076660.712959:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076660.712961:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616599296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076660.712980:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616599424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076660.713022:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076660.713031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076660/real 1769076660] req@ffff995b868c5500 x1855007616599424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076660.713039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616599424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076660.713045:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076660.713048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076660.713050:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616599424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076660.713054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616599552:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076660.713065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076660.713071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076660/real 1769076660] req@ffff995b868c6d80 x1855007616599552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076660.713083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616599552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076660.713089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076660.713091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076660.713093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616599552:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076660.713096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616599680:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076660.713106:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076660.713111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076660/real 1769076660] req@ffff995b868c4a80 x1855007616599680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076660.713119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616599680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076660.713124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076660.713126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076660.713128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616599680:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076660.713131:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616599808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076660.713140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076660.713145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076660/real 1769076660] req@ffff995b868c7800 x1855007616599808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076660.713152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616599808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076660.713162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076660.713164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076660.713166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616599808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076660.713169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616599936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076660.713178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076660.713183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076660/real 1769076660] req@ffff995b868c5c00 x1855007616599936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076660.713190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616599936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076660.713195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076660.713196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076660.713198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616599936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076661.416541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076661.416608:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076661.736518:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076661.736531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076661.736532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076661.736543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076662.440523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076662.440595:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076662.760546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076662.760546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076662.760551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076662.760565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076663.464541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076663.464635:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076663.784543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076663.784546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076663.784558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076663.784578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076664.488522:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076664.488582:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076664.808552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076664.808552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076664.808559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076664.808591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076665.512540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076665.512598:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076665.832542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076665.832543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076665.832567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076665.832594:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076665.832601:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076665.832606:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076665.832617:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3651 00000100:00080000:0.0:1769076665.832628:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076665.832636:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076665.832674:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616600064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076665.832703:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076665.832708:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076665.832710:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076665.832715:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3651 00000100:00080000:0.0:1769076665.832720:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076665.832726:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076665.832737:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076665.832740:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616600064:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076665.832757:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616600192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076665.832772:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076665.832776:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076665.832778:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076665.832782:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3651 00000100:00100000:3.0:1769076665.832837:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616600192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076665.832870:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076665.832958:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076665.832967:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076665.832974:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076665.832991:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616600320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076665.833013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076665.833017:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076665.833019:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076665.833024:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3651 00000100:00080000:0.0:1769076665.833029:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076665.833033:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076665.833039:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076665.833042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616600320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076665.833052:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616600448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076665.833066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076665.833070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076665.833072:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076665.833076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616600448:192.168.204.132@tcp:8:kworker.0 00000100:00080000:0.0:1769076665.833082:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3651 00000100:00080000:0.0:1769076665.833088:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076665.833094:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076665.833100:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076665.833113:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616600576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076665.833130:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076665.833134:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769076665.833143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616600576:192.168.204.132@tcp:8:kworker.0 00000100:00080000:0.0:1769076665.833144:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076665.833155:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3651 00000100:00080000:0.0:1769076665.833160:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076665.833165:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076665.833172:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076665.833189:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616600704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076665.833208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076665.833212:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076665.833214:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076665.833220:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3651 00000100:00080000:0.0:1769076665.833225:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076665.833227:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616600704:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076665.833233:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076665.833242:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076665.833264:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616600832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076665.833308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616600832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076666.536519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076666.536627:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076666.856543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076666.856560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076666.856629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076666.856631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076667.560559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076667.560626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076667.880539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076667.880545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076667.880547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076667.880612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076668.584525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076668.584607:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076668.904539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076668.904569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076668.904580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076668.904717:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769076668.904760:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769076668.904809:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076668.904832:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076668.904892:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076668.904924:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076668.904942:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076668.904945:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076665/real 1769076668] req@ffff995b84642a00 x1855007616600192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1769076668.904958:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076668.904977:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769076668.904999:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769076668.905002:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616600192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076668.905020:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076668.905027:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076668.905032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616600192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076668.905042:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076665/real 1769076668] req@ffff995b84643800 x1855007616600064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076668.905059:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616600064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076668.905069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076668.905073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076668.905078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616600064:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076668.905112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076665/real 1769076668] req@ffff995b84640700 x1855007616600832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076668.905129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616600832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076668.905138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076668.905142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076668.905146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616600832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076668.905155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076665/real 1769076668] req@ffff995b84642680 x1855007616600704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076668.905168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616600704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076668.905177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076668.905181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076668.905184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616600704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076668.905192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076665/real 1769076668] req@ffff995b84641c00 x1855007616600576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076668.905206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616600576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076668.905216:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076668.905220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076668.905224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616600576:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076668.905231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076665/real 1769076668] req@ffff995b84640000 x1855007616600448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076668.905243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616600448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076668.905252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076668.905256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076668.905259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616600448:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076668.905267:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076665/real 1769076668] req@ffff995b84640e00 x1855007616600320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076668.905280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616600320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076668.905292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076668.905295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076668.905299:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616600320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076669.608511:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076669.608583:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076669.928542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076669.928558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076669.928569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076669.928599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076670.632535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076670.632584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076670.952543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076670.952552:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076670.952558:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076670.952562:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076670.952564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076670.952570:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3656 00000100:00080000:2.0:1769076670.952581:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076670.952588:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076670.952607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076670.952618:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616600960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076670.952628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076670.952640:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076670.952646:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076670.952648:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076670.952654:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3656 00000100:00080000:2.0:1769076670.952658:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076670.952664:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076670.952674:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076670.952681:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616600960:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076670.952691:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616601088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076670.952705:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076670.952709:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076670.952711:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076670.952715:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3656 00000100:00080000:2.0:1769076670.952720:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076670.952724:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076670.952729:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076670.952743:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616601216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076670.952754:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076670.952755:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076670.952762:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076670.952766:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076670.952771:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3656 00000100:00080000:2.0:1769076670.952776:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076670.952780:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076670.952787:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076670.952805:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616601344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076670.952806:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076670/real 1769076670] req@ffff995b85acaa00 x1855007616600960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076670.952839:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076670.952844:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076670.952847:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076670.952859:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3656 00000100:00080000:2.0:1769076670.952865:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076670.952870:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076670.952876:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616600960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076670.952878:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769076670.952895:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076670.952897:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616601472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076670.952907:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076670.952913:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076670.952916:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616600960:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076670.952920:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076670.952925:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076670.952931:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3656 00000100:00080000:2.0:1769076670.952936:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076670.952940:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076670.952946:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076670.952952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076670.952962:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616601600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076670.952989:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076670.953000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076670.953004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076670.953006:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076670.953012:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3656 00000100:00100000:1.0:1769076670.953017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076670/real 1769076670] req@ffff995b85ac9880 x1855007616601088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076670.953018:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076670.953027:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076670.953038:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076670.953046:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616601088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076670.953055:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616601728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076670.953060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076670.953071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076670.953077:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076670.953087:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076670.953115:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076670.953128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076670/real 1769076670] req@ffff995b85ac9c00 x1855007616601216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076670.953139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616601216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076670.953159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076670.953163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076670.953167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076670.953175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601344:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076670.953201:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076670.953217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076670/real 1769076670] req@ffff995b85acb800 x1855007616601344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076670.953230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616601344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076670.953241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076670.953245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076670.953248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601344:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076670.953254:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601472:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076670.953276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076670.953288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076670/real 1769076670] req@ffff995b85aca300 x1855007616601472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076670.953301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616601472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076670.953310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076670.953314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076670.953318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601472:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076670.953342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616601600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076670.953367:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076670.953378:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076670/real 1769076670] req@ffff995b85ac8a80 x1855007616601600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076670.953390:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616601600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076670.953398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076670.953401:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076670.953404:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616601600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076670.953410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616601728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076670.953430:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076670.953443:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076670/real 1769076670] req@ffff995b85acb100 x1855007616601728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076670.953488:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616601728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076670.953499:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076670.953503:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076670.953506:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616601728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076671.656541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076671.656641:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076671.976712:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076671.976749:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076671.976764:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076671.976843:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076672.680527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076672.680584:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076673.000522:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076673.000548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076673.000582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076673.000606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076673.704524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076673.704619:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076674.024578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076674.024589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076674.024624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076674.024629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076674.728607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076674.728729:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076675.048637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076675.048637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076675.048705:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076675.048705:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076675.752571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076675.752686:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076676.072601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076676.072601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076676.072611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076676.072612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076676.072624:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076676.072629:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076676.072641:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3661 00000100:00080000:2.0:1769076676.072654:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076676.072663:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076676.072673:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076676.072721:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616601856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076676.072774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076676.072778:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076676.072779:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076676.072784:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3661 00000100:00080000:2.0:1769076676.072788:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076676.072792:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076676.072802:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076676.072816:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616601856:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076676.072817:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616601984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076676.072855:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076676.072861:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076676.072864:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076676.072870:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3661 00000100:00080000:2.0:1769076676.072874:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076676.072879:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076676.072886:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076676.072904:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616602112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076676.072916:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076676.072928:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076676.072932:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076676.072935:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076676.072941:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3661 00000100:00080000:2.0:1769076676.072948:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076676.072956:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076676.072960:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076676/real 1769076676] req@ffff995b85acbb80 x1855007616601856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076676.072967:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076676.072984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616601856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076676.072986:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616602240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076676.073040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076676.073050:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076676.073054:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076676.073059:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3661 00000100:00080000:1.0:1769076676.073060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076676.073069:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076676.073074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076676.073080:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076676.073088:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616601856:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769076676.073092:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076676.073116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076676.073125:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616602368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076676.073142:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076676.073150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076676.073154:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076676.073158:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076676.073164:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3661 00000100:00100000:1.0:1769076676.073168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076676/real 1769076676] req@ffff995b85ac8380 x1855007616601984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076676.073170:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076676.073183:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076676.073201:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076676.073204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616601984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076676.073217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076676.073222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076676.073228:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616602496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076676.073231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616601984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076676.073251:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616602112:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076676.073278:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076676.073279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076676.073288:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076676.073290:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076676.073298:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076676/real 1769076676] req@ffff995b85acb100 x1855007616602112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076676.073303:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3661 00000100:00080000:2.0:1769076676.073317:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076676.073328:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076676.073346:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076676.073353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616602112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076676.073368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076676.073368:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616602624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076676.073387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076676.073395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616602112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076676.073404:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616602240:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076676.073419:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076676.073426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076676/real 1769076676] req@ffff995b85ac8a80 x1855007616602240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076676.073437:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616602240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076676.073447:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076676.073451:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076676.073484:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616602240:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076676.073512:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616602368:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076676.073529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076676.073537:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076676/real 1769076676] req@ffff995b85aca300 x1855007616602368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076676.073546:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616602368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076676.073557:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076676.073560:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076676.073562:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616602368:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076676.073566:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616602496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076676.073577:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076676.073583:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076676/real 1769076676] req@ffff995b85acb800 x1855007616602496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076676.073590:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616602496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076676.073596:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076676.073598:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076676.073600:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616602496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076676.073604:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616602624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076676.073614:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076676.073619:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076676/real 1769076676] req@ffff995b85ac9c00 x1855007616602624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076676.073626:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616602624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076676.073632:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076676.073634:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076676.073636:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616602624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076676.776583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076676.776691:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076677.096610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076677.096620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076677.096621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076677.096748:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076677.800601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076677.800749:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076678.120551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076678.120551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076678.120613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076678.120627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076678.824725:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076678.824840:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076679.144581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076679.144594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076679.144664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076679.144686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076679.848640:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076679.848690:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076680.168563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076680.168598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076680.168635:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076680.168635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076680.872690:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076680.872715:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076681.192577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076681.192588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076681.192655:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076681.192658:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076681.192668:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076681.192697:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076681.192706:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076681.192717:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3666 00000100:00080000:2.0:1769076681.192733:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076681.192741:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076681.192781:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616602752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076681.192815:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076681.192821:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076681.192823:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076681.192828:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3666 00000100:00080000:2.0:1769076681.192833:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076681.192838:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076681.192848:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076681.192862:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616602880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076681.192875:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616602752:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076681.192876:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076681.192906:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076681.192907:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076681.192912:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3666 00000100:00080000:2.0:1769076681.192915:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076681.192918:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076681.192921:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076681.192933:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616603008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076681.192939:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076681.192942:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076681.192943:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076681.192947:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3666 00000100:00080000:2.0:1769076681.192950:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076681.192952:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076681.192956:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076681.192964:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616603136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076681.192970:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076681.192972:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076681.192973:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076681.192976:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3666 00000100:00080000:2.0:1769076681.192979:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076681.192982:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076681.192986:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076681.192993:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616603264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076681.193005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076681.193007:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076681.193015:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076681.193017:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076681.193020:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3666 00000100:00080000:2.0:1769076681.193024:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076681.193027:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076681.193031:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076681.193039:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616603392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076681.193046:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076681.193049:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076681.193051:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076681.193057:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3666 00000100:00080000:2.0:1769076681.193059:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076681.193062:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076681.193065:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076681.193074:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616603520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076681.193097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076681/real 1769076681] req@ffff995b85ac9880 x1855007616602752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076681.193137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616602752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076681.193151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076681.193157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076681.193162:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616602752:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076681.193239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616602880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076681.193272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076681.193290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076681/real 1769076681] req@ffff995b85acaa00 x1855007616602880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076681.193307:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616602880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076681.193318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076681.193348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076681.193352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616602880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076681.193361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076681.193385:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076681.193426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076681/real 1769076681] req@ffff995b85ac9c00 x1855007616603008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076681.193448:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616603008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076681.193540:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076681.193546:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076681.193551:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076681.193560:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603136:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076681.193596:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076681.193610:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076681/real 1769076681] req@ffff995b85acb800 x1855007616603136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076681.193626:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616603136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076681.193656:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076681.193662:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076681.193667:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603136:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076681.193675:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603264:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076681.193716:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076681.193730:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076681/real 1769076681] req@ffff995b85aca300 x1855007616603264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076681.193745:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616603264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076681.193768:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076681.193772:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076681.193775:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076681.193782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616603392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076681.193803:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076681.193814:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076681/real 1769076681] req@ffff995b85ac8a80 x1855007616603392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076681.193827:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616603392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076681.193838:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076681.193842:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076681.193846:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616603392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076681.193853:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616603520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076681.193871:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076681.193882:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076681/real 1769076681] req@ffff995b85acb100 x1855007616603520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076681.193894:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616603520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076681.193904:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076681.193919:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076681.193923:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616603520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076681.896599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076681.896761:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076682.216562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076682.216592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076682.216634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076682.216642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076682.920678:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076682.920724:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076683.240549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076683.240556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076683.240621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076683.240676:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076683.944544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076683.944637:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076684.264605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076684.264618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076684.264662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076684.264686:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076684.968614:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076684.968790:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076685.288653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076685.288667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076685.288675:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076685.288691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076685.992545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076685.992628:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076686.312644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076686.312671:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076686.312675:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076686.312690:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076686.312739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076686.312746:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076686.312749:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076686.312758:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3672 00000100:00080000:0.0:1769076686.312773:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076686.312779:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076686.312815:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616603648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076686.312842:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076686.312845:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076686.312847:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076686.312850:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3672 00000100:00080000:0.0:1769076686.312853:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076686.312856:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076686.312865:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076686.312877:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616603776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076686.312886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076686.312889:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076686.312890:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076686.312895:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3672 00000100:00080000:0.0:1769076686.312898:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076686.312901:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076686.312905:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076686.312922:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616603648:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076686.312924:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616603904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076686.312956:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076686.312958:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076686.312959:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076686.312962:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3672 00000100:00080000:0.0:1769076686.312964:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076686.312966:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076686.312970:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076686.312982:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616604032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076686.312992:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076686.312994:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076686.312996:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076686.312999:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3672 00000100:00080000:0.0:1769076686.313002:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076686.313005:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076686.313009:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076686.313017:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616604160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076686.313025:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076686.313027:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076686.313029:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076686.313032:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3672 00000100:00080000:0.0:1769076686.313075:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076686.313080:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076686.313086:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076686.313097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076686.313105:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616604288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076686.313116:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076686.313120:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076686.313121:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076686.313124:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3672 00000100:00080000:0.0:1769076686.313145:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076686.313152:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076686.313158:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076686.313187:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616604416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076686.313199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076686/real 1769076686] req@ffff995b84640380 x1855007616603648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076686.313244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616603648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076686.313261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076686.313268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076686.313274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616603648:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076686.313290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076686.313323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076686.313339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076686/real 1769076686] req@ffff995b84641f80 x1855007616603776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076686.313366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616603776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076686.313377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076686.313382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076686.313386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076686.313435:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076686.313532:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076686.313591:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076686/real 1769076686] req@ffff995b84640a80 x1855007616603904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076686.313611:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616603904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076686.313626:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076686.313630:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076686.313634:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616603904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076686.313644:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604032:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076686.313675:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076686.313689:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076686/real 1769076686] req@ffff995b84640e00 x1855007616604032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076686.313705:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616604032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076686.313717:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076686.313721:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076686.313724:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604032:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076686.313731:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076686.313750:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076686.313761:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076686/real 1769076686] req@ffff995b84640000 x1855007616604160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076686.313775:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616604160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076686.313787:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076686.313790:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076686.313794:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076686.313801:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616604288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076686.313820:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076686.313831:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076686/real 1769076686] req@ffff995b84641c00 x1855007616604288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076686.313845:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616604288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076686.313865:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076686.313869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076686.313873:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616604288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076686.313879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616604416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076686.313898:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076686.313909:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076686/real 1769076686] req@ffff995b84642680 x1855007616604416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076686.313922:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616604416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076686.313933:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076686.313937:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076686.313940:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616604416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076687.016635:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076687.016759:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076687.336673:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076687.336694:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076687.336752:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076687.336757:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076688.040659:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076688.040708:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076688.360628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076688.360657:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076688.360680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076688.360693:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076689.064541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076689.064650:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076689.384588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076689.384617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076689.384641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076689.384660:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076690.088593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076690.088635:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076690.408637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076690.408647:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076690.408747:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076690.408803:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076691.112605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076691.112739:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076691.432582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076691.432608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076691.432615:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076691.432619:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076691.432629:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3677 00000100:00100000:2.0:1769076691.432645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076691.432646:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076691.432651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076691.432661:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076691.432712:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616604544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076691.432729:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076691.432752:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076691.432759:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076691.432763:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076691.432771:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3677 00000100:00080000:0.0:1769076691.432776:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076691.432781:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076691.432788:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616604544:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076691.432791:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076691.432811:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616604672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076691.432824:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076691.432828:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076691.432829:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076691.432832:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3677 00000100:00080000:0.0:1769076691.432836:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076691.432839:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076691.432843:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076691.432851:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076691.432855:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616604800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076691.432868:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076691.432872:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076691.432874:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076691.432878:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3677 00000100:00080000:0.0:1769076691.432883:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076691.432888:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076691.432891:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076691/real 1769076691] req@ffff995b84640700 x1855007616604544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076691.432895:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076691.432911:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616604544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076691.432917:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616604928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076691.432921:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076691.432929:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076691.432930:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076691.432936:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769076691.432938:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616604544:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076691.432939:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076691.432948:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3677 00000100:00080000:0.0:1769076691.432952:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076691.432955:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076691.432959:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076691.432968:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076691.432969:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616605056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076691.432982:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076691.432992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076691.432994:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076691.432997:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076691.433001:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3677 00000100:00080000:0.0:1769076691.433004:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076691.433005:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076691/real 1769076691] req@ffff995b84643800 x1855007616604672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076691.433011:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076691.433019:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076691.433024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616604672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076691.433034:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616605184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076691.433036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076691.433046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076691.433048:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076691.433053:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076691.433055:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076691.433055:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604672:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076691.433063:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3677 00000100:00080000:0.0:1769076691.433066:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076691.433071:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076691.433072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604800:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769076691.433075:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076691.433087:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616605312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076691.433106:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076691.433122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076691/real 1769076691] req@ffff995b84642a00 x1855007616604800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076691.433136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616604800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076691.433147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076691.433151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076691.433153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076691.433158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604928:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076691.433174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076691.433182:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076691/real 1769076691] req@ffff995b84641180 x1855007616604928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076691.433203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616604928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076691.433209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076691.433212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076691.433214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616604928:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076691.433227:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605056:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076691.433244:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076691.433255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076691/real 1769076691] req@ffff995b84641500 x1855007616605056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076691.433267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616605056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076691.433277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076691.433281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076691.433284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076691.433290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616605184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076691.433307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076691.433315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076691/real 1769076691] req@ffff995b84643b80 x1855007616605184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076691.433322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616605184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076691.433333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076691.433335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076691.433337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616605184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076691.433341:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616605312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076691.433350:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076691.433356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076691/real 1769076691] req@ffff995b84641880 x1855007616605312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076691.433363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616605312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076691.433369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076691.433370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076691.433372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616605312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076692.136535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076692.136648:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076692.456587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076692.456598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076692.456626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076692.456648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076693.160600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076693.160708:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076693.480607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076693.480671:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076693.480680:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076693.480744:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076694.184574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076694.184749:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076694.504558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076694.504559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076694.504560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076694.504632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076695.208550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076695.208617:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076695.528559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076695.528575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076695.528575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076695.528606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076696.232546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076696.232616:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076696.552589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076696.552593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076696.552606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076696.552615:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076696.552620:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076696.552631:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3682 00000100:00100000:3.0:1769076696.552634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076696.552661:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076696.552672:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076696.552687:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076696.552718:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616605440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076696.552757:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076696.552763:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076696.552765:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076696.552770:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3682 00000100:00080000:0.0:1769076696.552774:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076696.552779:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076696.552790:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076696.552802:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616605440:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076696.552810:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616605568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076696.552834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076696.552837:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076696.552839:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076696.552843:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3682 00000100:00080000:0.0:1769076696.552846:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076696.552850:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076696.552854:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076696.552869:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616605696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076696.552883:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076696.552886:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076696.552894:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076696.552898:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076696.552906:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3682 00000100:00080000:0.0:1769076696.552912:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076696.552919:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076696.552927:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076696.552931:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076696/real 1769076696] req@ffff995b84643480 x1855007616605440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076696.552951:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616605824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076696.552962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616605440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076696.552964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076696.552973:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076696.552974:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076696.552979:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3682 00000100:00080000:0.0:1769076696.552987:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076696.552995:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076696.553001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769076696.553002:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769076696.553016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076696.553021:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616605952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076696.553029:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616605440:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076696.553036:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076696.553042:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076696.553044:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076696.553048:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3682 00000100:00080000:0.0:1769076696.553052:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076696.553055:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076696.553058:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076696.553069:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076696.553071:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616606080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076696.553084:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076696.553104:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076696.553106:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076696.553111:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3682 00000100:00080000:0.0:1769076696.553118:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076696.553123:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076696.553127:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076696.553136:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076696.553150:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616606208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076696.553157:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076696/real 1769076696] req@ffff995b84643100 x1855007616605568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076696.553224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616605568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076696.553253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076696.553259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076696.553275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076696.553285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076696.553315:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076696.553328:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076696/real 1769076696] req@ffff995b84642d80 x1855007616605696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076696.553339:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616605696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076696.553353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076696.553356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076696.553359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076696.553365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605824:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076696.553399:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076696.553412:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076696/real 1769076696] req@ffff995b84641880 x1855007616605824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076696.553435:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616605824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076696.553443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076696.553446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076696.553449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605824:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076696.553476:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605952:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076696.553516:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076696.553527:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076696/real 1769076696] req@ffff995b84643b80 x1855007616605952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076696.553538:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616605952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076696.553548:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076696.553551:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076696.553555:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616605952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076696.553574:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616606080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076696.553589:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076696.553598:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076696/real 1769076696] req@ffff995b84641500 x1855007616606080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076696.553609:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616606080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076696.553617:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076696.553620:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076696.553623:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616606080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076696.553628:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616606208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076696.553642:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076696.553667:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076696/real 1769076696] req@ffff995b84641180 x1855007616606208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076696.553676:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616606208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076696.553683:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076696.553685:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076696.553687:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616606208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076697.256545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076697.256576:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076697.576542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076697.576561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076697.576564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076697.576584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076698.280524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076698.280587:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076698.600515:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076698.600556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076698.600556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076698.600557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076699.304548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076699.304598:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076699.624560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076699.624560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076699.624569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076699.624599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076700.328586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076700.328687:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076700.648568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076700.648596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076700.648637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076700.648667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076701.352545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076701.352616:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076701.672590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076701.672591:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076701.672598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076701.672600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076701.672608:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076701.672624:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3687 00000100:00100000:3.0:1769076701.672624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076701.672643:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076701.672647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076701.672661:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076701.672710:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616606336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076701.672733:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076701.672736:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076701.672738:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076701.672742:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3687 00000100:00080000:1.0:1769076701.672746:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076701.672749:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076701.672757:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076701.672768:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616606464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076701.672781:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616606336:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076701.672784:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076701.672800:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076701.672803:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076701.672807:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3687 00000100:00080000:1.0:1769076701.672809:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076701.672811:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076701.672814:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076701.672826:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616606592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076701.672833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076701.672835:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076701.672837:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076701.672841:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3687 00000100:00080000:1.0:1769076701.672843:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076701.672846:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076701.672850:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076701.672859:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616606720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076701.672864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076701.672869:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076701.672871:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076701.672873:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3687 00000100:00080000:1.0:1769076701.672876:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076701.672878:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076701.672881:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076701.672887:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616606848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076701.672893:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076701.672895:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076701.672898:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076701.672901:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3687 00000100:00080000:1.0:1769076701.672906:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076701.672912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076701.672921:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076701.672930:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076701.672949:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616606976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076701.672962:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076701.672965:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076701.672967:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076701.672976:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3687 00000100:00080000:1.0:1769076701.672983:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076701.672986:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076701/real 1769076701] req@ffff995b868c6a00 x1855007616606336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076701.672995:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076701.673018:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076701.673032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616606336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076701.673043:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616607104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076701.673068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076701.673079:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076701.673104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616606336:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076701.673158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616606464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076701.673208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076701.673235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076701/real 1769076701] req@ffff995b868c7100 x1855007616606464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076701.673262:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616606464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076701.673277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076701.673282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076701.673286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616606464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076701.673328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616606592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076701.673359:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076701.673374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076701/real 1769076701] req@ffff995b868c6300 x1855007616606592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076701.673388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616606592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076701.673401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076701.673405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076701.673409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616606592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076701.673416:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616606720:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076701.673436:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076701.673446:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076701/real 1769076701] req@ffff995b868c5c00 x1855007616606720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076701.673494:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616606720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076701.673506:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076701.673510:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076701.673514:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616606720:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076701.673521:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616606848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076701.673543:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076701.673555:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076701/real 1769076701] req@ffff995b868c7800 x1855007616606848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076701.673570:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616606848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076701.673580:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076701.673583:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076701.673587:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616606848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076701.673593:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616606976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076701.673611:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076701.673622:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076701/real 1769076701] req@ffff995b868c4a80 x1855007616606976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076701.673635:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616606976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076701.673645:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076701.673648:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076701.673651:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616606976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076701.673658:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616607104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076701.673675:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076701.673685:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076701/real 1769076701] req@ffff995b868c6d80 x1855007616607104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076701.673697:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616607104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076701.673714:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076701.673718:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076701.673722:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616607104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076702.376614:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076702.376630:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076702.696539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076702.696539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076702.696599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076702.696673:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076703.400587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076703.400630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076703.720612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076703.720612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076703.720631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076703.720704:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076704.424617:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076704.424706:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076704.744594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076704.744607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076704.744610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076704.744660:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076705.448536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076705.448602:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076705.768578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076705.768578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076705.768578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076705.768592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076706.472588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076706.472749:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076706.792548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076706.792560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076706.792584:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076706.792591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076706.792595:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076706.792598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076706.792606:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3692 00000100:00080000:2.0:1769076706.792617:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076706.792625:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076706.792639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076706.792674:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616607232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076706.792701:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076706.792706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076706.792708:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076706.792713:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3692 00000100:00080000:2.0:1769076706.792717:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076706.792722:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076706.792725:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616607232:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769076706.792732:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076706.792747:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616607360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076706.792758:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076706.792763:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076706.792766:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076706.792771:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3692 00000100:00080000:2.0:1769076706.792788:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076706.792793:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076706.792799:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076706.792802:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076706.792813:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616607488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076706.792824:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076706.792828:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076706.792830:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076706.792835:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3692 00000100:00080000:2.0:1769076706.792840:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076706.792845:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076706.792848:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076706/real 1769076706] req@ffff995b85ac8380 x1855007616607232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076706.792851:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076706.792862:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616607616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076706.792868:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616607232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076706.792872:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076706.792876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076706.792878:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076706.792880:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076706.792883:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3692 00000100:00080000:3.0:1769076706.792885:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076706.792887:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076706.792890:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616607232:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076706.792891:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076706.792897:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076706.792908:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616607744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076706.792916:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616607360:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076706.792917:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076706.792920:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076706.792922:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076706.792926:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3692 00000100:00080000:2.0:1769076706.792929:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076706.792933:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076706.792937:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076706.792938:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076706.792947:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616607872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076706.792951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076706/real 1769076706] req@ffff995b85acbb80 x1855007616607360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076706.792958:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076706.792961:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076706.792962:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076706.792966:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3692 00000100:00080000:2.0:1769076706.792969:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076706.792973:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076706.792978:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076706.792987:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616608000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076706.793260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616607360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076706.793272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076706.793277:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076706.793284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616607360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076706.793290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616607488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076706.793315:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076706.793332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076706/real 1769076706] req@ffff995b85acb100 x1855007616607488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076706.793341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616607488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076706.793426:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076706.793430:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076706.793433:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616607488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076706.793438:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616607616:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076706.793482:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076706.793494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076706/real 1769076706] req@ffff995b85ac8a80 x1855007616607616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076706.793504:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616607616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076706.793515:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076706.793518:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076706.793520:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616607616:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076706.793532:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616607744:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076706.793544:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076706.793551:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076706/real 1769076706] req@ffff995b85aca300 x1855007616607744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076706.793559:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616607744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076706.793565:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076706.793567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076706.793568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616607744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076706.793572:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616607872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076706.793582:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076706.793588:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076706/real 1769076706] req@ffff995b85acb800 x1855007616607872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076706.793596:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616607872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076706.793601:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076706.793603:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076706.793605:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616607872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076706.793609:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616608000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076706.793618:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076706.793624:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076706/real 1769076706] req@ffff995b85ac9c00 x1855007616608000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076706.793630:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616608000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076706.793635:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076706.793637:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076706.793639:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616608000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076707.496513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076707.496584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076707.816547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076707.816555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076707.816559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076707.816599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076708.520562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076708.520665:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076708.840570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076708.840572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076708.840584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076708.840599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076709.544591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076709.544674:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076709.864576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076709.864612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076709.864620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076709.864671:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076710.568557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076710.568611:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076710.888555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076710.888559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076710.888593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076710.888632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076711.592618:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076711.592737:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076711.912628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076711.912681:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076711.912694:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076711.912702:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076711.912719:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3697 00000100:00100000:3.0:1769076711.912759:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076711.912765:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076711.912782:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076711.912788:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076711.912800:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076711.912843:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616608128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076711.912893:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076711.912902:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076711.912907:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076711.912917:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3697 00000100:00080000:2.0:1769076711.912927:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076711.912936:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076711.912952:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076711.912971:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616608128:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076711.912984:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616608256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076711.913011:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076711.913018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076711.913023:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076711.913031:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3697 00000100:00080000:2.0:1769076711.913053:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076711.913066:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076711.913068:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076711.913083:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076711.913109:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616608384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076711.913169:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076711/real 1769076711] req@ffff995b85acaa00 x1855007616608128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076711.913182:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076711.913194:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076711.913199:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076711.913224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616608128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076711.913231:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3697 00000100:00080000:2.0:1769076711.913248:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076711.913264:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076711.913271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076711.913278:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769076711.913282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076711.913293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616608128:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076711.913314:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616608512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076711.913328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616608256:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076711.913353:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076711.913361:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076711.913366:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076711.913394:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3697 00000800:00000100:3.0:1769076711.913396:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076711.913407:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076711.913419:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076711.913424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076711/real 1769076711] req@ffff995b85ac9880 x1855007616608256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076711.913433:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076711.913446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616608256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076711.913460:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076711.913464:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616608640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076711.913469:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076711.913481:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616608256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076711.913523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616608384:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076711.913549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076711.913555:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076711.913565:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076711.913570:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076711.913580:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076711/real 1769076711] req@ffff995b85ac9c00 x1855007616608384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076711.913585:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3697 00000100:00080000:2.0:1769076711.913601:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076711.913610:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616608384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076711.913618:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076711.913638:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076711.913640:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076711.913651:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076711.913656:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616608384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076711.913673:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616608768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076711.913683:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616608512:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769076711.913712:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076711.913722:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076711.913724:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076711.913732:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076711.913741:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076711/real 1769076711] req@ffff995b85acb800 x1855007616608512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076711.913744:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3697 00000100:00080000:2.0:1769076711.913757:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076711.913769:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076711.913779:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076711.913780:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616608512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076711.913805:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076711.913808:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616608896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076711.913819:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076711.913836:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616608512:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076711.913858:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616608640:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076711.913896:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076711.913914:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076711/real 1769076711] req@ffff995b85aca300 x1855007616608640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076711.913929:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616608640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076711.913939:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076711.913943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076711.913947:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616608640:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076711.913964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616608768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076711.913982:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076711.913993:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076711/real 1769076711] req@ffff995b85ac8a80 x1855007616608768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076711.914007:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616608768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076711.914035:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076711.914038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076711.914042:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616608768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076711.914049:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616608896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076711.914069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076711.914082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076711/real 1769076711] req@ffff995b85acb100 x1855007616608896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076711.914096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616608896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076711.914106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076711.914109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076711.914113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616608896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076712.616616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076712.616672:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076712.936583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076712.936584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076712.936588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076712.936621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076713.640549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076713.640630:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076713.960593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076713.960593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076713.960649:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076713.960667:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076714.664531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076714.664596:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076714.984623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076714.984624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076714.984624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076714.984658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076715.688559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076715.688621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076716.008566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076716.008568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076716.008574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076716.008592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076716.712610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076716.712684:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076717.032615:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076717.032630:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769076717.032636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076717.032638:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076717.032638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076717.032643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076717.032662:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3702 00000100:00080000:0.0:1769076717.032679:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076717.032690:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076717.032734:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076717.032744:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616609024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076717.032798:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076717.032807:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076717.032811:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076717.032822:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3702 00000100:00080000:0.0:1769076717.032831:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076717.032841:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076717.032858:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616609024:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076717.032859:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076717.032894:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616609152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076717.032912:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076717.032918:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076717.032921:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076717.032926:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3702 00000100:00080000:0.0:1769076717.032933:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076717.032957:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769076717.032959:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769076717.032967:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076717.032992:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616609280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076717.033006:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076717.033011:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076717.033014:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076717.033023:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3702 00000100:00100000:3.0:1769076717.033026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076717/real 1769076717] req@ffff995b84642a00 x1855007616609024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076717.033029:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076717.033093:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076717.033102:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076717.033105:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616609024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076717.033122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769076717.033124:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616609408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076717.033136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076717.033150:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076717.033152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616609024:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076717.033159:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076717.033166:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076717.033172:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3702 00000100:00080000:0.0:1769076717.033178:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076717.033182:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076717.033189:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076717.033196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616609152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076717.033205:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616609536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076717.033222:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076717.033226:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076717.033229:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076717.033236:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3702 00000800:00000100:3.0:1769076717.033245:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076717.033248:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076717.033256:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076717.033263:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076717.033273:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076717/real 1769076717] req@ffff995b84643800 x1855007616609152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076717.033286:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616609664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076717.033298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616609152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076717.033323:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076717.033326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076717.033338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076717.033340:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076717.033347:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616609152:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076717.033349:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076717.033361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616609280:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076717.033365:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3702 00000100:00080000:0.0:1769076717.033378:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076717.033387:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769076717.033395:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769076717.033398:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076717.033413:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076717/real 1769076717] req@ffff995b84640700 x1855007616609280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076717.033422:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616609792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076717.033439:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616609280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076717.033499:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076717.033505:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076717.033510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616609280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076717.033518:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616609408:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076717.033540:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076717.033553:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076717/real 1769076717] req@ffff995b84642680 x1855007616609408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076717.033566:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616609408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076717.033577:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076717.033581:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076717.033584:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616609408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076717.033602:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616609536:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076717.033618:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076717.033628:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076717/real 1769076717] req@ffff995b84641c00 x1855007616609536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076717.033641:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616609536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076717.033650:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076717.033654:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076717.033658:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616609536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076717.033664:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616609664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076717.033681:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076717.033710:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076717/real 1769076717] req@ffff995b84640000 x1855007616609664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076717.033724:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616609664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076717.033734:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076717.033738:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076717.033742:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616609664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076717.033749:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616609792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076717.033768:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076717.033778:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076717/real 1769076717] req@ffff995b84640e00 x1855007616609792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076717.033792:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616609792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076717.033802:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076717.033805:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076717.033809:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616609792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076717.736553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076717.736704:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076718.056588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076718.056605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076718.056650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076718.056683:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076718.760581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076718.760722:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076719.080636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076719.080639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076719.080647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076719.080649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076719.784534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076719.784579:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076720.104533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076720.104534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076720.104544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076720.104571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076720.808552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076720.808670:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076721.128589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076721.128617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076721.128633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076721.128727:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076721.832573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076721.832667:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076722.152648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076722.152702:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076722.152734:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076722.152744:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076722.152751:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076722.152757:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076722.152770:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3707 00000100:00080000:1.0:1769076722.152788:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076722.152801:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076722.152810:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076722.152857:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616609920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076722.152895:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076722.152900:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076722.152902:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076722.152908:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3707 00000100:00080000:1.0:1769076722.152913:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076722.152919:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076722.152931:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076722.152949:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616610048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076722.152986:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616609920:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076722.152998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076722.153005:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076722.153008:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076722.153015:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3707 00000100:00080000:1.0:1769076722.153022:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076722.153028:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076722.153036:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076722.153055:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616610176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076722.153066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076722.153070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076722.153073:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076722.153079:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3707 00000100:00080000:1.0:1769076722.153084:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076722.153088:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076722.153114:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076722.153114:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076722.153137:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616610304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076722.153149:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076722.153153:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076722.153156:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076722.153162:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3707 00000100:00080000:1.0:1769076722.153168:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076722.153172:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076722.153179:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076722.153180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076722/real 1769076722] req@ffff995b868c5500 x1855007616609920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076722.153198:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616610432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076722.153213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616609920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076722.153250:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076722.153252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076722.153263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076722.153264:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769076722.153272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616609920:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076722.153275:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076722.153290:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3707 00000100:00080000:1.0:1769076722.153298:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076722.153306:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076722.153310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610048:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769076722.153323:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076722.153357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076722.153359:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616610560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076722.153375:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076722/real 1769076722] req@ffff995b868c4700 x1855007616610048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076722.153386:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076722.153398:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076722.153399:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616610048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076722.153406:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076722.153421:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076722.153428:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3707 00000100:00080000:3.0:1769076722.153433:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076722.153442:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610048:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076722.153444:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076722.153452:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610176:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076722.153462:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076722.153519:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076722.153526:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076722.153550:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616610688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076722.153553:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076722/real 1769076722] req@ffff995b868c7480 x1855007616610176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076722.153616:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616610176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076722.153632:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076722.153636:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076722.153642:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076722.153650:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610304:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076722.153671:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076722.153682:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076722/real 1769076722] req@ffff995b868c6d80 x1855007616610304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076722.153696:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616610304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076722.153705:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076722.153708:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076722.153712:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610304:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076722.153718:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610432:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076722.153734:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076722.153744:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076722/real 1769076722] req@ffff995b868c4a80 x1855007616610432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076722.153756:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616610432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076722.153767:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076722.153770:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076722.153774:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610432:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076722.153793:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616610560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076722.153810:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076722.153820:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076722/real 1769076722] req@ffff995b868c7800 x1855007616610560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076722.153832:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616610560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076722.153842:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076722.153845:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076722.153849:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616610560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076722.153856:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616610688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076722.153871:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076722.153881:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076722/real 1769076722] req@ffff995b868c5c00 x1855007616610688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076722.153893:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616610688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076722.153903:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076722.153906:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076722.153910:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616610688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076722.856514:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076722.856607:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076723.176574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076723.176575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076723.176609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076723.176658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076723.880589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076723.880690:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076724.200585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076724.200635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076724.200640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076724.200651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076724.904536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076724.904636:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076725.224569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076725.224585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076725.224604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076725.224616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076725.928566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076725.928749:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076726.248561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076726.248571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076726.248650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076726.248692:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076726.952545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076726.952592:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076727.272517:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076727.272535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076727.272556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076727.272564:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076727.272568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076727.272574:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076727.272580:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076727.272589:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3712 00000100:00080000:0.0:1769076727.272602:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076727.272610:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076727.272647:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616610816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076727.272678:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076727.272682:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076727.272684:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076727.272688:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3712 00000100:00080000:0.0:1769076727.272692:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076727.272698:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076727.272708:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076727.272721:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616610816:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076727.272725:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616610944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076727.272740:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076727.272744:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076727.272746:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076727.272750:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3712 00000100:00080000:0.0:1769076727.272754:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076727.272758:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076727.272763:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076727.272777:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616611072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076727.272787:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076727.272790:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076727.272794:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076727.272798:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076727.272804:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3712 00000100:00080000:0.0:1769076727.272810:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076727.272815:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076727.272821:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076727.272828:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076727/real 1769076727] req@ffff995b84640a80 x1855007616610816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076727.272837:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616611200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076727.272847:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616610816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076727.272852:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076727.272861:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076727.272881:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076727.272883:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076727.272889:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3712 00000100:00080000:3.0:1769076727.272894:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076727.272896:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076727.272903:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076727.272910:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616610816:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076727.272911:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076727.272940:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076727.272943:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616611328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076727.272958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076727.272963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1769076727.272964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076727.272968:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076727.272976:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3712 00000100:00100000:3.0:1769076727.272980:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076727/real 1769076727] req@ffff995b84641f80 x1855007616610944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076727.272982:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076727.272992:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076727.272997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616610944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076727.273000:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076727.273009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076727.273013:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076727.273016:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616610944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076727.273022:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616611456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076727.273023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611072:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076727.273039:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076727.273045:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076727.273048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076727.273052:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076727.273056:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076727/real 1769076727] req@ffff995b84640380 x1855007616611072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076727.273061:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3712 00000100:00080000:0.0:1769076727.273070:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076727.273075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616611072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076727.273080:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076727.273087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769076727.273090:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076727.273093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076727.273098:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076727.273109:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616611584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076727.273114:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611200:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076727.273137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076727.273145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076727/real 1769076727] req@ffff995b84640e00 x1855007616611200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076727.273155:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616611200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076727.273161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076727.273164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076727.273166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611200:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076727.273177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611328:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076727.273188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076727.273194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076727/real 1769076727] req@ffff995b84640000 x1855007616611328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076727.273201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616611328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076727.273208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076727.273210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076727.273212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611328:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076727.273215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616611456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076727.273225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076727.273230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076727/real 1769076727] req@ffff995b84641c00 x1855007616611456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076727.273237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616611456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076727.273243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076727.273245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076727.273246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616611456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076727.273250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616611584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076727.273258:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076727.273264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076727/real 1769076727] req@ffff995b84642680 x1855007616611584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076727.273276:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616611584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076727.273281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076727.273282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076727.273284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616611584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076727.976631:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076727.976700:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076728.296614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076728.296636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076728.296675:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076728.296747:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076729.000575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076729.000694:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076729.320601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076729.320668:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076729.320689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076729.320694:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076730.024656:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076730.024772:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076730.344586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076730.344651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076730.344653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076730.344664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076731.048625:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076731.048716:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076731.368596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076731.368651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076731.368652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076731.368696:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076732.072604:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076732.072729:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076732.392543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076732.392599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076732.392631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076732.392637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076732.392642:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076732.392656:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3718 00000100:00100000:3.0:1769076732.392663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076732.392666:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076732.392671:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076732.392686:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076732.392726:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616611712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076732.392753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076732.392766:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076732.392769:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076732.392774:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3718 00000100:00080000:1.0:1769076732.392799:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076732.392805:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076732.392813:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616611712:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076732.392817:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076732.392836:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616611840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076732.392848:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076732.392852:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076732.392854:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076732.392857:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3718 00000100:00080000:1.0:1769076732.392863:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076732.392865:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076732.392870:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076732.392883:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616611968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076732.392891:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076732.392894:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076732.392896:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076732.392900:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3718 00000100:00080000:1.0:1769076732.392903:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076732.392906:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076732.392911:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076732.392925:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616612096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076732.392933:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076732.392936:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076732.392939:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076732.392946:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3718 00000100:00100000:3.0:1769076732.392948:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611840:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076732.392949:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076732.392959:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076732.392965:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076732.392978:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616612224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076732.392987:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076732.392991:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076732.392993:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076732.392997:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3718 00000100:00080000:1.0:1769076732.393001:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076732.393003:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611968:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076732.393011:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076732.393019:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076732.393034:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616612352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076732.393042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612096:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769076732.393047:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076732.393052:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076732.393055:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076732.393063:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3718 00000100:00080000:1.0:1769076732.393070:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076732.393074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612224:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769076732.393082:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076732.393094:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076732.393101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616612352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076732.393115:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616612480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076732.393168:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616612480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076733.096531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076733.096645:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076733.416540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076733.416595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076733.416683:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076733.416723:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076734.120542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076734.120606:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076734.440558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076734.440577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076734.440589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076734.440629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076735.144626:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076735.144688:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076735.464580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076735.464581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076735.464646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076735.464815:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:3.0:1769076735.464871:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:3.0:1769076735.464938:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769076735.464959:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769076735.465007:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769076735.465020:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769076735.465036:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769076735.465048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076732/real 1769076735] req@ffff995b868c7100 x1855007616611840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1769076735.465054:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769076735.465071:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769076735.465089:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769076735.465090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616611840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076735.465111:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076735.465120:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076735.465126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076735.465138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076732/real 1769076735] req@ffff995b868c6a00 x1855007616611968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076735.465171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616611968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076735.465189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076735.465194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076735.465198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616611968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076735.465207:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076732/real 1769076735] req@ffff995b868c5c00 x1855007616612096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076735.465221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616612096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076735.465231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076735.465235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076735.465239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612096:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076735.465246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076732/real 1769076735] req@ffff995b868c6300 x1855007616611712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076735.465259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616611712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076735.465267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076735.465270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076735.465273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616611712:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076735.465311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076732/real 1769076735] req@ffff995b868c6d80 x1855007616612480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076735.465325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616612480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076735.465334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076735.465337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076735.465340:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616612480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076735.465348:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076732/real 1769076735] req@ffff995b868c7800 x1855007616612224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076735.465360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616612224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076735.465370:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076735.465374:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076735.465377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612224:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076735.465384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076732/real 1769076735] req@ffff995b868c4a80 x1855007616612352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076735.465396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616612352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076735.465404:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076735.465408:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076735.465411:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616612352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076736.168543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076736.168698:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076736.488635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076736.488669:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076736.488686:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076736.488751:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076737.192574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076737.192644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076737.512590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076737.512611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076737.512611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076737.512670:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076737.512676:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769076737.512679:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076737.512681:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076737.512711:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3723 00000100:00080000:0.0:1769076737.512755:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076737.512762:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076737.512795:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616612608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076737.512838:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076737.512843:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076737.512845:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076737.512849:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3723 00000100:00080000:0.0:1769076737.512853:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076737.512858:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076737.512867:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076737.512883:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616612736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076737.512891:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616612608:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076737.512900:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076737.512905:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076737.512908:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076737.512912:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3723 00000100:00080000:0.0:1769076737.512916:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076737.512919:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076737.512923:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076737.512935:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616612864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076737.512945:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076737.512949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076737.512951:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076737.512956:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3723 00000100:00080000:0.0:1769076737.512961:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076737.512968:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076737.512970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769076737.512975:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076737.512990:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616612992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076737.513001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076737.513004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076737.513007:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076737.513012:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3723 00000100:00080000:0.0:1769076737.513017:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076737.513022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076737/real 1769076737] req@ffff995b84640700 x1855007616612608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076737.513026:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076737.513035:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076737.513058:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616613120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076737.513065:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616612608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076737.513086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076737.513090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076737.513092:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076737.513098:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076737.513100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076737.513107:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3723 00000100:00100000:2.0:1769076737.513110:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616612608:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076737.513114:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076737.513122:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076737.513129:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076737.513143:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616613248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076737.513145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612736:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076737.513164:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076737.513168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076737.513170:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076737.513176:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3723 00000100:00080000:0.0:1769076737.513181:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076737.513185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076737.513190:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076737.513198:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076737.513207:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076737/real 1769076737] req@ffff995b84643800 x1855007616612736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076737.513214:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616613376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076737.513232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616612736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076737.513243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076737.513247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076737.513251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076737.513259:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076737.513283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076737.513295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076737/real 1769076737] req@ffff995b84642a00 x1855007616612864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076737.513314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616612864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076737.513326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076737.513329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076737.513332:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076737.513339:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612992:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076737.513358:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076737.513368:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076737/real 1769076737] req@ffff995b84641180 x1855007616612992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076737.513381:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616612992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076737.513389:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076737.513393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076737.513396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616612992:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076737.513403:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616613120:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076737.513420:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076737.513431:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076737/real 1769076737] req@ffff995b84641500 x1855007616613120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076737.513449:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616613120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076737.513491:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076737.513494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076737.513498:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616613120:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076737.513520:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616613248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076737.513543:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076737.513554:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076737/real 1769076737] req@ffff995b84643b80 x1855007616613248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076737.513565:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616613248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076737.513573:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076737.513577:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076737.513580:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616613248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076737.513585:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616613376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076737.513601:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076737.513611:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076737/real 1769076737] req@ffff995b84641880 x1855007616613376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076737.513622:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616613376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076737.513629:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076737.513632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076737.513636:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616613376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076738.216578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076738.216702:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076738.536545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076738.536566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076738.536605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076738.536715:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076739.240561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076739.240630:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076739.560529:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076739.560534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076739.560554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076739.560554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076740.264606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076740.264712:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076740.584539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076740.584552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076740.584552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076740.584559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076741.288572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076741.288696:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076741.608608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076741.608613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076741.608656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076741.608658:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076742.312543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076742.312595:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076742.632562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076742.632563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076742.632570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076742.632575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076742.632641:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076742.632646:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076742.632661:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076742.632669:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3728 00000100:00080000:2.0:1769076742.632679:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076742.632686:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076742.632734:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616613504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076742.632763:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076742.632768:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076742.632771:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076742.632776:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3728 00000100:00080000:2.0:1769076742.632781:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076742.632782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616613504:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076742.632791:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076742.632803:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076742.632821:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616613632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076742.632831:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076742.632834:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076742.632836:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076742.632840:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3728 00000100:00080000:2.0:1769076742.632845:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076742.632849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076742.632851:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076742.632859:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076742.632888:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616613760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076742.632898:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076742/real 1769076742] req@ffff995b85acbb80 x1855007616613504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076742.632901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076742.632913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076742.632916:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076742.632921:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3728 00000100:00080000:2.0:1769076742.632925:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076742.632929:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076742.632934:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616613504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076742.632936:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769076742.632949:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076742.632956:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616613888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076742.632956:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076742.632964:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616613504:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076742.632978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076742.632982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076742.632985:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076742.632990:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3728 00000100:00080000:2.0:1769076742.633020:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076742.633025:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076742.633030:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076742.633052:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616614016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076742.633077:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076742.633080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616613632:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076742.633081:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076742.633085:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076742.633090:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3728 00000100:00080000:2.0:1769076742.633093:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076742.633096:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076742.633101:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076742.633106:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076742.633119:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616614144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076742.633120:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076742/real 1769076742] req@ffff995b85ac8380 x1855007616613632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076742.633145:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076742.633150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616613632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076742.633151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076742.633155:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076742.633161:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3728 00000100:00080000:2.0:1769076742.633163:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076742.633165:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076742.633168:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076742.633174:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616614272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076742.633258:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076742.633263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076742.633268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616613632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076742.633320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616613760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076742.633367:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076742.633377:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076742/real 1769076742] req@ffff995b85acb100 x1855007616613760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076742.633390:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616613760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076742.633401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076742.633405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076742.633409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616613760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076742.633415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616613888:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076742.633436:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076742.633447:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076742/real 1769076742] req@ffff995b85ac8a80 x1855007616613888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076742.633486:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616613888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076742.633498:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076742.633502:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076742.633505:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616613888:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076742.633521:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614016:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076742.633541:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076742.633553:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076742/real 1769076742] req@ffff995b85aca300 x1855007616614016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076742.633566:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616614016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076742.633572:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076742.633574:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076742.633577:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614016:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076742.633581:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616614144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076742.633595:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076742.633602:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076742/real 1769076742] req@ffff995b85acb800 x1855007616614144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076742.633629:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616614144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076742.633651:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076742.633653:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076742.633655:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616614144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076742.633659:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616614272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076742.633673:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076742.633680:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076742/real 1769076742] req@ffff995b85ac9c00 x1855007616614272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076742.633687:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616614272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076742.633693:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076742.633694:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076742.633696:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616614272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076743.336587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076743.336672:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076743.656511:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076743.656533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076743.656540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076743.656549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769076744.360611:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076744.360615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076744.680511:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076744.680558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076744.680676:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076744.680680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076745.384598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076745.384665:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076745.704504:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076745.704544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076745.704554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076745.704578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076746.408586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076746.408614:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076746.728511:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076746.728553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076746.728572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076746.728587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076747.432571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076747.432626:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076747.752583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076747.752605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076747.752605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076747.752652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076747.752652:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076747.752663:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076747.752670:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076747.752682:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3733 00000100:00080000:3.0:1769076747.752697:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076747.752705:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076747.752757:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616614400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076747.752788:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076747.752793:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076747.752795:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076747.752800:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3733 00000100:00080000:3.0:1769076747.752804:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076747.752809:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076747.752821:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076747.752833:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616614400:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076747.752837:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616614528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076747.752856:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076747.752860:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076747.752862:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076747.752867:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3733 00000100:00080000:3.0:1769076747.752871:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076747.752874:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076747.752879:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076747.752891:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616614656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076747.752902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076747.752906:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076747.752908:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076747.752912:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3733 00000100:00080000:3.0:1769076747.752917:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076747.752921:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076747.752927:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076747.752933:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076747.752941:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616614784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076747.752953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076747.752957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076747.752959:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076747.752963:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3733 00000100:00080000:3.0:1769076747.752967:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076747.752971:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076747.752977:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076747.752984:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076747/real 1769076747] req@ffff995b84558000 x1855007616614400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076747.752991:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616614912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076747.753014:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076747.753018:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076747.753020:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076747.753024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616614400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076747.753027:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3733 00000100:00080000:3.0:1769076747.753035:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076747.753043:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076747.753046:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769076747.753049:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076747.753056:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076747.753064:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616615040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076747.753077:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076747.753078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616614400:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076747.753091:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076747.753097:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076747.753102:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3733 00000100:00080000:3.0:1769076747.753107:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076747.753111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614528:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076747.753115:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076747.753124:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076747.753147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076747.753147:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616615168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076747.753165:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076747/real 1769076747] req@ffff995b84559500 x1855007616614528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076747.753182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616614528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076747.753192:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076747.753196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076747.753200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076747.753207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076747.753223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076747.753233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076747/real 1769076747] req@ffff995b84558a80 x1855007616614656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076747.753254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616614656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076747.753265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076747.753269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076747.753273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076747.753278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614784:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076747.753311:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076747.753323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076747/real 1769076747] req@ffff995b8455a680 x1855007616614784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076747.753336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616614784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076747.753353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076747.753360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076747.753364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614784:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076747.753370:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614912:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076747.753387:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076747.753398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076747/real 1769076747] req@ffff995b8455b100 x1855007616614912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076747.753410:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616614912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076747.753428:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076747.753431:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076747.753435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616614912:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076747.753441:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616615040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076747.753481:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076747.753494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076747/real 1769076747] req@ffff995b8455aa00 x1855007616615040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076747.753507:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616615040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076747.753517:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076747.753533:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076747.753537:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616615040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076747.753557:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616615168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076747.753578:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076747.753589:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076747/real 1769076747] req@ffff995b8455bb80 x1855007616615168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076747.753602:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616615168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076747.753611:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076747.753615:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076747.753619:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616615168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076748.456559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076748.456636:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076748.776553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076748.776554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076748.776615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076748.776620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076749.480553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076749.480618:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076749.800529:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076749.800626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076749.800700:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076749.800732:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076750.504545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076750.504594:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076750.824602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076750.824614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076750.824646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076750.824657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076751.528540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076751.528601:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076751.848602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076751.848612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076751.848632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076751.848636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076752.552532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076752.552619:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076752.872569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076752.872577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076752.872577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076752.872581:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076752.872591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076752.872599:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076752.872610:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3738 00000100:00080000:3.0:1769076752.872623:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076752.872632:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076752.872658:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076752.872676:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616615296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076752.872713:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076752.872717:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076752.872719:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076752.872724:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3738 00000100:00080000:3.0:1769076752.872730:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076752.872735:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076752.872745:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076752.872753:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616615296:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076752.872760:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616615424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076752.872773:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076752.872777:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076752.872779:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076752.872783:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3738 00000100:00080000:3.0:1769076752.872788:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076752.872793:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076752.872798:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076752.872814:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076752.872815:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616615552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076752.872827:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076752.872831:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076752.872833:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076752.872837:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3738 00000100:00080000:3.0:1769076752.872842:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076752.872848:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076752.872850:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076752/real 1769076752] req@ffff995b8455bb80 x1855007616615296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076752.872854:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076752.872870:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616615296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076752.872875:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616615680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076752.872886:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076752.872892:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076752.872895:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076752.872898:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616615296:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076752.872902:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076752.872907:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076752.872912:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3738 00000100:00080000:3.0:1769076752.872916:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076752.872920:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076752.872925:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076752.872927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616615424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076752.872942:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616615808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076752.872952:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076752.872962:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076752.872966:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076752/real 1769076752] req@ffff995b8455aa00 x1855007616615424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076752.872968:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076752.872972:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076752.872981:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3738 00000100:00100000:2.0:1769076752.872984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616615424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076752.872989:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076752.872993:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076752.872998:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076752.872999:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076752.873007:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616615424:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076752.873009:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076752.873014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616615552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076752.873025:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616615936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076752.873032:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076752.873043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076752/real 1769076752] req@ffff995b8455b100 x1855007616615552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076752.873062:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076752.873070:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076752.873072:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076752.873073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616615552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076752.873082:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3738 00000100:00080000:3.0:1769076752.873090:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076752.873091:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076752.873096:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076752.873099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076752.873104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616615552:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076752.873105:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076752.873110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616615680:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076752.873121:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616616064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076752.873135:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076752.873150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076752/real 1769076752] req@ffff995b8455a680 x1855007616615680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076752.873164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616615680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076752.873175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076752.873178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076752.873181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616615680:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076752.873192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616615808:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076752.873207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076752.873215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076752/real 1769076752] req@ffff995b84558a80 x1855007616615808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076752.873223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616615808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076752.873229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076752.873231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076752.873233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616615808:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076752.873236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616615936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076752.873246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076752.873251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076752/real 1769076752] req@ffff995b84559500 x1855007616615936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076752.873259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616615936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076752.873264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076752.873266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076752.873268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616615936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076752.873271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616616064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076752.873279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076752.873285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076752/real 1769076752] req@ffff995b84558000 x1855007616616064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076752.873292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616616064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076752.873298:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076752.873300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076752.873302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616616064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076753.576558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076753.576647:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076753.896562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076753.896580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076753.896597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076753.896607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076754.600542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076754.600613:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076754.920551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076754.920572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076754.920579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076754.920608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076755.624568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076755.624602:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076755.944592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076755.944592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076755.944671:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076755.944672:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076756.648552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076756.648624:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076756.968574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076756.968598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076756.968644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076756.968696:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076757.672549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076757.672566:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076757.992548:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076757.992557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076757.992558:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076757.992563:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076757.992575:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3743 00000100:00100000:0.0:1769076757.992582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076757.992586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076757.992588:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076757.992600:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076757.992600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076757.992641:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616616192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076757.992664:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076757.992669:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076757.992672:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076757.992675:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3743 00000100:00080000:2.0:1769076757.992678:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076757.992682:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616616192:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076757.992685:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076757.992696:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076757.992707:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616616320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076757.992716:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076757.992718:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076757.992720:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076757.992723:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3743 00000100:00080000:2.0:1769076757.992733:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076757.992736:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076757.992739:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076757.992749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076757.992750:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616616448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076757.992758:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076757.992761:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076757.992762:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076757.992765:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3743 00000100:00080000:2.0:1769076757.992768:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076757.992770:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076757.992774:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076757.992782:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616616576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076757.992788:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076757.992791:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076757.992792:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076757.992794:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3743 00000100:00080000:2.0:1769076757.992797:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076757.992801:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076757/real 1769076757] req@ffff995b85ac9880 x1855007616616192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076757.992802:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076757.992809:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076757.992821:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616616704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076757.992828:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076757.992830:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076757.992831:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076757.992835:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3743 00000100:00100000:3.0:1769076757.992836:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616616192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076757.992837:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076757.992844:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076757.992851:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076757.992856:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076757.992861:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616616832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076757.992864:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076757.992873:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616616192:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076757.992875:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076757.992880:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076757.992882:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076757.992885:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3743 00000100:00080000:2.0:1769076757.992888:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076757.992890:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076757.992893:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076757.992899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616616320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076757.992902:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616616960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076757.992925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076757.992943:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076757/real 1769076757] req@ffff995b85acaa00 x1855007616616320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076757.992957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616616320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076757.992966:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076757.992970:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076757.992974:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616616320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076757.992980:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616616448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076757.993000:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076757.993009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076757/real 1769076757] req@ffff995b85ac9c00 x1855007616616448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076757.993017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616616448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076757.993024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076757.993034:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076757.993036:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616616448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076757.993040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616616576:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076757.993053:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076757.993060:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076757/real 1769076757] req@ffff995b85acb800 x1855007616616576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076757.993067:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616616576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076757.993073:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076757.993076:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076757.993078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616616576:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076757.993081:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616616704:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076757.993090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076757.993095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076757/real 1769076757] req@ffff995b85aca300 x1855007616616704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076757.993102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616616704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076757.993108:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076757.993110:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076757.993112:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616616704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076757.993115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616616832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076757.993124:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076757.993130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076757/real 1769076757] req@ffff995b85ac8a80 x1855007616616832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076757.993137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616616832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076757.993143:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076757.993145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076757.993147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616616832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076757.993159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616616960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076757.993170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076757.993181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076757/real 1769076757] req@ffff995b85acb100 x1855007616616960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076757.993193:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616616960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076757.993201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076757.993205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076757.993209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616616960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076758.696558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076758.696633:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076759.016559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076759.016574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076759.016603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076759.016622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076759.720521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076759.720597:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076760.040579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076760.040596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076760.040622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076760.040626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076760.744536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076760.744603:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076761.064559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076761.064570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076761.064620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076761.064634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076761.768535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076761.768601:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076762.088572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076762.088575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076762.088603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076762.088630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076762.792529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076762.792581:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076763.112569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076763.112593:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076763.112602:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076763.112609:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076763.112611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076763.112614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076763.112621:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3748 00000100:00080000:2.0:1769076763.112638:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076763.112649:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076763.112650:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076763.112701:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616617088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076763.112731:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076763.112736:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076763.112739:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076763.112747:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3748 00000100:00080000:2.0:1769076763.112752:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076763.112763:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076763.112768:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616617088:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769076763.112778:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076763.112797:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616617216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076763.112810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076763.112814:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076763.112816:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076763.112823:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3748 00000100:00080000:2.0:1769076763.112827:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076763.112831:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076763.112837:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076763.112849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076763.112852:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616617344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076763.112865:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076763.112870:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076763.112872:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076763.112887:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3748 00000100:00080000:2.0:1769076763.112892:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076763.112897:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076763.112904:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076763/real 1769076763] req@ffff995b85ac8380 x1855007616617088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076763.112907:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076763.112926:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616617472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076763.112936:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616617088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076763.112941:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076763.112946:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076763.112948:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076763.112954:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3748 00000100:00080000:2.0:1769076763.112963:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076763.112975:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076763.112977:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076763.112982:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769076763.112987:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076763.112996:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616617088:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076763.113001:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616617600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076763.113015:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076763.113019:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076763.113021:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076763.113028:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3748 00000100:00100000:0.0:1769076763.113032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616617216:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076763.113035:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076763.113047:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076763.113054:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076763.113068:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616617728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076763.113075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076763.113097:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076763.113102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076763.113107:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076763.113108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076763/real 1769076763] req@ffff995b85acbb80 x1855007616617216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076763.113118:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3748 00000100:00080000:2.0:1769076763.113126:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076763.113133:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076763.113141:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076763.113146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616617216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076763.113159:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616617856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076763.113163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076763.113181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076763.113196:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616617216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076763.113207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616617344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076763.113230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076763.113243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076763/real 1769076763] req@ffff995b85acb100 x1855007616617344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076763.113256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616617344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076763.113280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076763.113284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076763.113288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616617344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076763.113296:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616617472:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076763.113318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076763.113329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076763/real 1769076763] req@ffff995b85ac8a80 x1855007616617472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076763.113342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616617472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076763.113352:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076763.113356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076763.113359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616617472:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076763.113375:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616617600:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076763.113395:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076763.113406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076763/real 1769076763] req@ffff995b85aca300 x1855007616617600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076763.113418:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616617600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076763.113428:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076763.113432:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076763.113435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616617600:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076763.113482:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616617728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076763.113503:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076763.113515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076763/real 1769076763] req@ffff995b85acb800 x1855007616617728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076763.113529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616617728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076763.113538:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076763.113542:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076763.113545:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616617728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076763.113552:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616617856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076763.113570:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076763.113581:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076763/real 1769076763] req@ffff995b85ac9c00 x1855007616617856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076763.113595:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616617856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076763.113605:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076763.113609:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076763.113613:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616617856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076763.816512:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076763.816567:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076764.136523:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076764.136529:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076764.136571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076764.136579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076764.840527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076764.840606:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076765.160557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076765.160576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076765.160585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076765.160601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076765.864575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076765.864695:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076766.184609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076766.184645:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076766.184684:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076766.184698:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076766.888532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076766.888639:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076767.208648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076767.208662:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076767.208665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076767.208667:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076767.912564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076767.912644:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076768.232636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076768.232642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076768.232645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076768.232645:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076768.232687:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076768.232694:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076768.232699:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076768.232712:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3753 00000100:00080000:3.0:1769076768.232727:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076768.232736:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076768.232783:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616617984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076768.232812:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076768.232816:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076768.232818:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076768.232822:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3753 00000100:00080000:3.0:1769076768.232826:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076768.232831:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076768.232841:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076768.232855:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616618112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076768.232861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616617984:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076768.232883:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076768.232887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076768.232889:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076768.232894:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3753 00000100:00080000:3.0:1769076768.232897:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076768.232901:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076768.232907:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076768.232921:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616618240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076768.232931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076768.232934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076768.232937:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076768.232941:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3753 00000100:00080000:3.0:1769076768.232944:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076768.232949:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076768.232956:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076768.232961:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076768.232969:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616618368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076768.232980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076768.232984:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076768.232986:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076768.233002:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3753 00000100:00080000:3.0:1769076768.233006:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076768.233009:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076768.233014:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076768.233023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076768/real 1769076768] req@ffff995b8455b480 x1855007616617984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076768.233036:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616618496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076768.233047:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076768.233049:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616617984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076768.233052:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076768.233058:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076768.233074:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3753 00000100:00080000:0.0:1769076768.233077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076768.233082:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076768.233086:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076768.233098:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616617984:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076768.233099:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076768.233114:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076768.233140:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616618112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076768.233148:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616618624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076768.233163:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076768.233167:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076768.233170:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076768.233208:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3753 00000100:00080000:3.0:1769076768.233213:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076768.233217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076768.233218:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076768.233241:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076768.233263:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616618752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076768.233264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076768/real 1769076768] req@ffff995b84558700 x1855007616618112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076768.233314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616618112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076768.233325:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076768.233329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076768.233333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616618112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076768.233340:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616618240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076768.233362:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076768.233372:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076768/real 1769076768] req@ffff995b84559c00 x1855007616618240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076768.233383:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616618240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076768.233392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076768.233395:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076768.233398:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616618240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076768.233403:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616618368:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076768.233422:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076768.233430:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076768/real 1769076768] req@ffff995b8455a300 x1855007616618368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076768.233440:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616618368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076768.233449:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076768.233475:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076768.233480:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616618368:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076768.233486:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616618496:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076768.233514:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076768.233525:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076768/real 1769076768] req@ffff995b8455ad80 x1855007616618496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076768.233536:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616618496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076768.233545:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076768.233547:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076768.233550:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616618496:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076768.233568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616618624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076768.233584:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076768.233593:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076768/real 1769076768] req@ffff995b84559880 x1855007616618624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076768.233604:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616618624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076768.233612:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076768.233615:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076768.233617:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616618624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076768.233622:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616618752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076768.233635:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076768.233644:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076768/real 1769076768] req@ffff995b8455b480 x1855007616618752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076768.233655:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616618752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076768.233663:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076768.233665:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076768.233668:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616618752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076768.936580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076768.936712:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076769.256593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076769.256606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076769.256672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076769.256702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076769.960593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076769.960620:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076770.280547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076770.280556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076770.280583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076770.280613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076770.984567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076770.984679:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076771.304561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076771.304609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076771.304625:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076771.304627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076772.008581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076772.008686:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076772.328536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076772.328542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076772.328547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076772.328570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076773.032525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076773.032602:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076773.352577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076773.352580:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076773.352582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076773.352588:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076773.352593:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076773.352599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076773.352604:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3759 00000100:00100000:0.0:1769076773.352612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076773.352616:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076773.352624:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076773.352662:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616618880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076773.352689:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076773.352693:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076773.352695:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076773.352710:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3759 00000100:00080000:1.0:1769076773.352715:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076773.352719:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076773.352730:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076773.352736:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616618880:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076773.352742:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616619008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076773.352757:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076773.352760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076773.352762:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076773.352766:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3759 00000100:00080000:1.0:1769076773.352769:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076773.352772:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076773.352777:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076773.352785:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616619136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076773.352801:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076773.352804:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076773.352806:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076773.352807:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076773.352809:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3759 00000100:00080000:1.0:1769076773.352811:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076773.352815:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076773.352818:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076773.352828:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616619264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076773.352835:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076773.352838:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076773.352840:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076773.352844:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3759 00000100:00100000:0.0:1769076773.352845:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076773/real 1769076773] req@ffff995b868c7480 x1855007616618880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076773.352847:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076773.352851:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076773.352856:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076773.352862:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616618880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076773.352864:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616619392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076773.352873:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076773.352874:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076773.352877:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076773.352878:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076773.352879:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076773.352882:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616618880:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076773.352883:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3759 00000100:00080000:1.0:1769076773.352886:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076773.352889:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076773.352893:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076773.352900:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616619520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076773.352905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619008:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076773.352909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076773.352913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076773.352914:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076773.352930:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3759 00000100:00080000:1.0:1769076773.352934:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076773.352937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076773.352937:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076773.352942:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076773.352949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076773/real 1769076773] req@ffff995b868c4700 x1855007616619008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076773.352950:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616619648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076773.352962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616619008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076773.352971:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076773.352974:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076773.352977:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076773.352984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076773.353003:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076773.353013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076773/real 1769076773] req@ffff995b868c5500 x1855007616619136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076773.353021:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616619136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076773.353027:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076773.353029:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076773.353032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076773.353036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619264:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076773.353048:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076773.353053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076773/real 1769076773] req@ffff995b868c4a80 x1855007616619264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076773.353061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616619264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076773.353066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076773.353068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076773.353070:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076773.353073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619392:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076773.353082:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076773.353088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076773/real 1769076773] req@ffff995b868c7800 x1855007616619392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076773.353121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616619392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076773.353127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076773.353129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076773.353131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076773.353143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616619520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076773.353157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076773.353167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076773/real 1769076773] req@ffff995b868c6d80 x1855007616619520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076773.353187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616619520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076773.353196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076773.353199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076773.353203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616619520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076773.353209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616619648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076773.353230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076773.353248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076773/real 1769076773] req@ffff995b868c6300 x1855007616619648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076773.353257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616619648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076773.353262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076773.353264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076773.353266:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616619648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076774.056552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076774.056619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076774.376586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076774.376586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076774.376592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076774.376624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076775.080552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076775.080611:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076775.400564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076775.400567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076775.400583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076775.400585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076776.104537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076776.104628:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076776.424566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076776.424566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076776.424568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076776.424594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076777.128592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076777.128672:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076777.448595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076777.448597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076777.448676:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076777.448700:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076778.152541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076778.152619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076778.472537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076778.472542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076778.472714:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076778.472725:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076778.472740:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076778.472751:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3764 00000100:00080000:3.0:1769076778.472764:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076778.472772:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076778.472772:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076778.472829:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616619776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076778.472858:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076778.472865:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076778.472872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076778.472885:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076778.472890:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3764 00000100:00100000:2.0:1769076778.472896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616619776:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076778.472899:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076778.472906:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076778.472916:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076778.472934:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616619904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076778.472949:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076778.472953:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076778.472955:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076778.472958:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3764 00000100:00080000:3.0:1769076778.472971:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076778.472976:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076778.472983:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076778.472989:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076778.473059:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616620032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076778.473061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076778/real 1769076778] req@ffff995b84558000 x1855007616619776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076778.473084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076778.473089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076778.473092:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076778.473098:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3764 00000100:00100000:2.0:1769076778.473102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616619776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076778.473107:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076778.473119:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076778.473122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769076778.473128:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769076778.473136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076778.473145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616619776:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076778.473159:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616620160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076778.473177:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076778.473181:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076778.473185:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076778.473195:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3764 00000100:00100000:2.0:1769076778.473199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619904:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076778.473203:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076778.473213:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076778.473222:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076778.473244:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616620288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076778.473261:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076778.473263:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076778.473302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076778/real 1769076778] req@ffff995b84559500 x1855007616619904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076778.473334:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616619904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076778.473343:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076778.473346:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076778.473348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076778.473352:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076778.473355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616619904:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076778.473365:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3764 00000100:00100000:2.0:1769076778.473366:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076778.473396:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076778.473414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076778/real 1769076778] req@ffff995b84558a80 x1855007616620032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076778.473423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616620032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076778.473431:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076778.473434:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076778.473436:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076778.473442:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076778.473502:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076778.473526:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076778/real 1769076778] req@ffff995b8455a680 x1855007616620160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076778.473549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616620160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076778.473565:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076778.473567:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076778.473577:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076778.473580:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076778.473599:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620288:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076778.473619:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076778.473628:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076778.473629:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076778.473651:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616620416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076778.473653:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076778/real 1769076778] req@ffff995b8455b100 x1855007616620288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076778.473687:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076778.473709:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076778.473712:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076778.473718:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616620288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076778.473719:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3764 00000100:00080000:3.0:1769076778.473726:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076778.473735:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076778.473744:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076778.473748:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076778.473756:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616620544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076778.473760:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076778.473769:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076778.473793:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616620416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076778.473818:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076778.473829:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076778/real 1769076778] req@ffff995b8455aa00 x1855007616620416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076778.473839:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616620416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076778.473846:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076778.473849:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076778.473852:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616620416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076778.473858:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616620544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076778.473870:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076778.473877:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076778/real 1769076778] req@ffff995b8455bb80 x1855007616620544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076778.473905:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616620544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076778.473913:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076778.473916:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076778.473918:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616620544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076779.176576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076779.176637:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076779.496552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076779.496553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076779.496553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076779.496596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076780.200561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076780.200663:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076780.520559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076780.520568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076780.520603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076780.520631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076781.224533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076781.224607:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076781.544533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076781.544544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076781.544547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076781.544562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076782.248536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076782.248586:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076782.568550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076782.568558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076782.568568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076782.568569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076783.272578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076783.272647:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076783.592543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076783.592543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076783.592557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076783.592624:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076783.592629:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076783.592633:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076783.592643:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3769 00000100:00080000:2.0:1769076783.592654:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076783.592661:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076783.592671:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076783.592701:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616620672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076783.592730:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076783.592733:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076783.592735:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076783.592739:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3769 00000100:00080000:2.0:1769076783.592741:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076783.592745:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076783.592752:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076783.592765:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616620672:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076783.592765:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616620800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076783.592779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076783.592782:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076783.592783:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076783.592786:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3769 00000100:00080000:2.0:1769076783.592789:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076783.592792:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076783.592795:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076783.592806:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616620928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076783.592813:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076783.592815:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076783.592817:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076783.592820:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3769 00000100:00080000:2.0:1769076783.592823:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076783.592826:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076783.592830:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076783.592834:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076783.592843:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616621056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076783.592850:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076783.592853:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076783.592854:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076783.592856:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3769 00000100:00080000:2.0:1769076783.592859:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076783.592861:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076783.592865:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076783.592876:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616621184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076783.592877:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076783/real 1769076783] req@ffff995b85acaa00 x1855007616620672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076783.592899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076783.592902:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076783.592904:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076783.592908:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3769 00000100:00080000:2.0:1769076783.592912:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076783.592913:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616620672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076783.592933:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076783.592944:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076783.592946:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076783.592959:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076783.592962:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616621312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076783.592966:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616620672:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076783.592988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076783.592991:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076783.592994:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620800:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076783.592996:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076783.593004:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3769 00000100:00080000:2.0:1769076783.593008:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076783.593012:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076783.593016:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076783.593021:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076783.593032:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616621440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076783.593035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076783/real 1769076783] req@ffff995b85ac9880 x1855007616620800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076783.593063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616620800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076783.593070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076783.593073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076783.593077:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076783.593082:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076783.593094:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076783.593101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076783/real 1769076783] req@ffff995b85ac9c00 x1855007616620928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076783.593109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616620928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076783.593116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076783.593118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076783.593120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616620928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076783.593124:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621056:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076783.593135:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076783.593141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076783/real 1769076783] req@ffff995b85acb800 x1855007616621056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076783.593149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616621056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076783.593156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076783.593158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076783.593160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076783.593163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621184:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076783.593173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076783.593179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076783/real 1769076783] req@ffff995b85aca300 x1855007616621184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076783.593187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616621184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076783.593194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076783.593196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076783.593198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076783.593201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616621312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076783.593211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076783.593217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076783/real 1769076783] req@ffff995b85ac8a80 x1855007616621312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076783.593225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616621312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076783.593231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076783.593232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076783.593234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616621312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076783.593247:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616621440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076783.593257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076783.593263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076783/real 1769076783] req@ffff995b85acb100 x1855007616621440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076783.593271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616621440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076783.593277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076783.593279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076783.593281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616621440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076784.296618:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076784.296664:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076784.616595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076784.616614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076784.616634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076784.616708:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076785.320540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076785.320734:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076785.640573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076785.640576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076785.640589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076785.640646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076786.344585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076786.344633:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076786.664672:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076786.664726:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076786.664737:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076786.664781:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076787.368561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076787.368657:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076787.688617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076787.688621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076787.688641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076787.688708:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076788.392549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076788.392651:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076788.712673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076788.712674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076788.712717:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076788.712729:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769076788.712733:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076788.712737:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076788.712755:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3774 00000100:00080000:2.0:1769076788.712770:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076788.712782:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076788.712789:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076788.712858:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616621568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076788.712904:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076788.712909:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076788.712912:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076788.712917:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3774 00000100:00080000:2.0:1769076788.712921:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076788.712927:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076788.712950:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076788.712967:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616621696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076788.712984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616621568:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076788.712993:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076788.712999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076788.713002:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076788.713007:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3774 00000100:00080000:2.0:1769076788.713012:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076788.713016:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076788.713022:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076788.713037:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616621824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076788.713049:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076788.713053:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076788.713055:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076788.713058:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3774 00000100:00080000:2.0:1769076788.713062:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076788.713066:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076788.713073:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076788.713090:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616621952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076788.713091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076788.713108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076788.713113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076788.713116:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076788.713121:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3774 00000100:00080000:2.0:1769076788.713126:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076788.713130:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076788.713135:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076788.713151:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616622080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076788.713161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076788/real 1769076788] req@ffff995b85acbb80 x1855007616621568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076788.713168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076788.713181:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076788.713187:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076788.713201:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3774 00000100:00080000:2.0:1769076788.713206:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076788.713214:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076788.713215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616621568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076788.713221:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076788.713238:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616622208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076788.713239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076788.713261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076788.713272:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076788.713274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616621568:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076788.713279:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076788.713286:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076788.713292:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3774 00000100:00080000:2.0:1769076788.713297:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076788.713301:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076788.713306:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076788.713316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076788.713323:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616622336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076788.713360:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076788.713381:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076788/real 1769076788] req@ffff995b85ac8380 x1855007616621696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076788.713396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616621696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076788.713406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076788.713410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076788.713414:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076788.713421:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076788.713439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076788.713450:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076788/real 1769076788] req@ffff995b85acb100 x1855007616621824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076788.713506:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616621824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076788.713518:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076788.713522:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076788.713526:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076788.713533:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621952:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076788.713555:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076788.713567:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076788/real 1769076788] req@ffff995b85ac8a80 x1855007616621952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076788.713580:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616621952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076788.713590:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076788.713594:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076788.713598:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616621952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076788.713604:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622080:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076788.713620:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076788.713633:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076788/real 1769076788] req@ffff995b85aca300 x1855007616622080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076788.713649:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616622080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076788.713665:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076788.713671:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076788.713678:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076788.713689:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616622208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076788.713733:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076788.713751:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076788/real 1769076788] req@ffff995b85acb800 x1855007616622208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076788.713765:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616622208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076788.713775:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076788.713779:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076788.713783:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616622208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076788.713808:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616622336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076788.713830:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076788.713875:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076788/real 1769076788] req@ffff995b85ac9c00 x1855007616622336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076788.713889:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616622336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076788.713899:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076788.713902:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076788.713906:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616622336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076789.416555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076789.416625:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076789.736556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076789.736585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076789.736607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076789.736638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076790.440556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076790.440612:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076790.760587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076790.760592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076790.760596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076790.760638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076791.464581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076791.464646:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076791.784587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076791.784622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076791.784632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076791.784681:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076792.488592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076792.488676:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076792.808542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076792.808558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076792.808590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076792.808600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076793.512568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076793.512710:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076793.832563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076793.832636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076793.832658:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076793.832672:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076793.832680:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076793.832696:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076793.832699:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3779 00000100:00100000:3.0:1769076793.832723:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076793.832732:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076793.832744:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076793.832834:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616622464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076793.832866:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076793.832871:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076793.832873:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076793.832878:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3779 00000100:00080000:1.0:1769076793.832883:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076793.832889:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076793.832901:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076793.832914:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616622592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076793.832930:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616622464:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076793.832933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076793.832949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076793.832952:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076793.832958:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3779 00000100:00080000:1.0:1769076793.832962:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076793.832967:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076793.832971:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076793.832985:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616622720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076793.832996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076793.833000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076793.833002:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076793.833005:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3779 00000100:00080000:1.0:1769076793.833016:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076793.833020:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076793.833025:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076793.833033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076793.833043:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616622848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076793.833055:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076793.833058:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076793.833061:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076793.833066:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3779 00000100:00080000:1.0:1769076793.833072:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076793.833078:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076793/real 1769076793] req@ffff995b868c5c00 x1855007616622464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076793.833083:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076793.833104:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076793.833114:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616622464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076793.833125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076793.833134:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616622976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076793.833141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076793.833148:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616622464:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076793.833179:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076793.833184:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769076793.833187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622592:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076793.833191:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076793.833203:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3779 00000100:00080000:1.0:1769076793.833214:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076793.833223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076793.833230:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076793.833237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076793/real 1769076793] req@ffff995b868c6a00 x1855007616622592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076793.833240:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076793.833265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616622592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076793.833271:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616623104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076793.833297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076793.833307:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076793.833309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076793.833323:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076793.833332:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076793.833333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622592:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076793.833340:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3779 00000100:00080000:1.0:1769076793.833353:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076793.833357:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076793.833364:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622720:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769076793.833365:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076793.833381:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616623232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076793.833409:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076793.833436:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076793/real 1769076793] req@ffff995b868c7100 x1855007616622720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076793.833474:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616622720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076793.833483:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076793.833486:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076793.833489:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076793.833495:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076793.833512:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076793.833521:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076793/real 1769076793] req@ffff995b868c6300 x1855007616622848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076793.833529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616622848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076793.833535:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076793.833537:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076793.833540:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076793.833544:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076793.833553:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076793.833559:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076793/real 1769076793] req@ffff995b868c6d80 x1855007616622976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076793.833566:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616622976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076793.833572:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076793.833574:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076793.833576:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616622976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076793.833592:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616623104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076793.833601:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076793.833607:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076793/real 1769076793] req@ffff995b868c7800 x1855007616623104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076793.833615:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616623104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076793.833621:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076793.833623:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076793.833624:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616623104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076793.833628:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616623232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076793.833637:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076793.833643:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076793/real 1769076793] req@ffff995b868c4a80 x1855007616623232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076793.833651:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616623232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076793.833667:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076793.833669:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076793.833671:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616623232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076794.536540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076794.536664:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076794.856587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076794.856637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076794.856648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076794.856651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076795.560565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076795.560634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076795.880542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076795.880600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076795.880609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076795.880669:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076796.584546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076796.584594:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076796.904569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076796.904611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076796.904621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076796.904717:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076797.608593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076797.608730:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076797.928587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076797.928592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076797.928632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076797.928677:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076798.632555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076798.632615:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076798.952583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076798.952586:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076798.952592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076798.952595:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076798.952603:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076798.952613:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3784 00000100:00080000:1.0:1769076798.952623:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076798.952632:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076798.952654:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076798.952692:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616623360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076798.952739:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076798.952744:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076798.952746:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076798.952752:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3784 00000100:00080000:1.0:1769076798.952757:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076798.952762:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076798.952772:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076798.952788:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616623488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076798.952792:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076798.952803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076798.952808:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076798.952810:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076798.952814:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3784 00000100:00080000:1.0:1769076798.952816:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076798.952820:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076798.952824:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076798.952833:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616623616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076798.952842:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076798.952845:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076798.952847:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076798.952851:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3784 00000100:00080000:1.0:1769076798.952853:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076798.952856:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076798.952860:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076798.952862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616623360:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076798.952873:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616623744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076798.952883:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076798.952885:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076798.952887:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076798.952890:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3784 00000100:00080000:1.0:1769076798.952893:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076798.952896:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076798.952900:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076798.952909:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616623872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076798.952917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076798.952920:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076798.952922:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076798.952925:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3784 00000100:00080000:1.0:1769076798.952929:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076798.952932:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076798.952936:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076798.952946:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616624000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076798.952963:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076798.952967:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616623488:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076798.952978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076798.952982:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076798.952989:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3784 00000100:00080000:1.0:1769076798.952996:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076798.953002:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616623616:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076798.953004:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076798.953017:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076798.953039:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616624128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076798.953040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616623744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076798.953069:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616623872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076798.953094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616624000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076798.953113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616624128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076799.656566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076799.656634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076799.976536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076799.976547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076799.976575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076799.976599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076800.680552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076800.680685:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076801.000563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076801.000564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076801.000593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076801.000625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076801.704574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076801.704645:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076802.024558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076802.024597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076802.024673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076802.024699:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:1.0:1769076802.024930:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076802.024987:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076802.025012:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076802.025078:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076802.025109:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076802.025122:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076802.025135:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076802.025165:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076802.025177:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076802.025278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076798/real 1769076802] req@ffff995b868c6300 x1855007616624128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076802.025311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616624128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076802.025328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076802.025335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076802.025341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616624128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076802.025351:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076798/real 1769076802] req@ffff995b868c4a80 x1855007616623744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076802.025370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616623744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076802.025384:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076802.025390:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076802.025395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616623744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076802.025406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076798/real 1769076802] req@ffff995b868c7800 x1855007616623872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076802.025423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616623872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076802.025436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076802.025441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076802.025446:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616623872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076802.025946:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076798/real 1769076802] req@ffff995b868c6d80 x1855007616624000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076802.025971:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616624000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076802.025986:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076802.025992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076802.025998:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616624000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076802.026009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076798/real 1769076802] req@ffff995b868c5500 x1855007616623360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076802.026039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616623360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076802.026053:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076802.026058:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076802.026064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616623360:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076802.026074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076798/real 1769076802] req@ffff995b868c4700 x1855007616623488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076802.026094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616623488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076802.026107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076802.026113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076802.026118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616623488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076802.026129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076798/real 1769076802] req@ffff995b868c7480 x1855007616623616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076802.026148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616623616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076802.026162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076802.026168:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076802.026173:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616623616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076802.728581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076802.728721:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076803.048584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076803.048585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076803.048614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076803.048656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076803.752537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076803.752625:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076804.072523:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076804.072528:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076804.072549:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076804.072555:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076804.072559:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076804.072567:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3789 00000100:00100000:0.0:1769076804.072591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076804.072592:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076804.072612:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076804.072633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076804.072645:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616624256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076804.072680:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076804.072683:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076804.072684:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076804.072688:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3789 00000100:00080000:1.0:1769076804.072690:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076804.072693:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076804.072699:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616624256:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076804.072701:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076804.072712:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616624384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076804.072720:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076804.072723:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076804.072725:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076804.072728:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3789 00000100:00080000:1.0:1769076804.072730:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076804.072732:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076804.072745:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076804.072752:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616624512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076804.072759:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076804.072761:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076804.072763:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076804.072765:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3789 00000100:00080000:1.0:1769076804.072768:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076804.072772:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769076804.072774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769076804.072777:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076804.072787:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616624640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076804.072793:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076804.072796:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076804.072797:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076804.072799:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3789 00000100:00080000:1.0:1769076804.072801:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076804.072804:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076804.072819:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076804.072825:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616624768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076804.072830:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076804.072834:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076804.072836:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076804.072842:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076804/real 1769076804] req@ffff995b868c7100 x1855007616624256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076804.072843:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3789 00000100:00080000:1.0:1769076804.072849:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076804.072857:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076804.072866:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076804.072879:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616624256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076804.072883:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616624896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076804.072904:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076804.072907:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076804.072914:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076804.072914:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076804.072919:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076804.072924:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616624256:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076804.072929:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3789 00000100:00080000:1.0:1769076804.072936:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076804.072940:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076804.072946:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076804.072959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616624384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076804.072960:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616625024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076804.073055:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076804.073071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076804/real 1769076804] req@ffff995b868c6a00 x1855007616624384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076804.073085:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616624384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076804.073095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076804.073099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076804.073103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616624384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076804.073110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616624512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076804.073132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076804.073149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076804/real 1769076804] req@ffff995b868c5c00 x1855007616624512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076804.073161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616624512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076804.073172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076804.073175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076804.073178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616624512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076804.073184:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616624640:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076804.073206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076804.073218:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076804/real 1769076804] req@ffff995b868c7480 x1855007616624640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076804.073231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616624640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076804.073241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076804.073245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076804.073249:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616624640:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076804.073255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616624768:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076804.073274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076804.073286:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076804/real 1769076804] req@ffff995b868c4700 x1855007616624768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076804.073300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616624768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076804.073322:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076804.073326:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076804.073330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616624768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076804.073336:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616624896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076804.073362:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076804.073374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076804/real 1769076804] req@ffff995b868c5500 x1855007616624896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076804.073385:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616624896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076804.073392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076804.073395:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076804.073398:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616624896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076804.073419:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616625024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076804.073439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076804.073451:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076804/real 1769076804] req@ffff995b868c6d80 x1855007616625024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076804.073489:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616625024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076804.073505:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076804.073508:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076804.073511:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616625024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076804.776539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076804.776617:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076805.096600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076805.096669:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076805.096698:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076805.096702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076805.800517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076805.800592:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076806.120575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076806.120575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076806.120576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076806.120594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076806.824559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076806.824623:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076807.144609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076807.144609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076807.144622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076807.144691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076807.848578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076807.848713:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076808.168658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076808.168660:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076808.168667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076808.168742:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076808.872732:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076808.872733:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076809.192584:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076809.192585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076809.192601:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076809.192609:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076809.192620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076809.192623:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3794 00000100:00100000:2.0:1769076809.192627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076809.192642:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076809.192660:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076809.192679:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076809.192711:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616625152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076809.192739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076809.192743:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076809.192744:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076809.192748:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3794 00000100:00080000:0.0:1769076809.192751:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076809.192755:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076809.192766:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076809.192780:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616625280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076809.192790:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616625152:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076809.192800:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076809.192806:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076809.192809:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076809.192815:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3794 00000100:00080000:0.0:1769076809.192819:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076809.192841:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076809.192847:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076809.192865:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616625408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076809.192875:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076809.192878:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076809.192880:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076809.192885:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3794 00000100:00080000:0.0:1769076809.192892:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076809.192896:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076809.192901:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076809.192909:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076809.192921:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616625536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076809.192933:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076809.192936:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076809.192938:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076809.192942:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3794 00000100:00080000:0.0:1769076809.192946:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076809.192950:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076809.192962:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076809.192963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076809/real 1769076809] req@ffff995b84642d80 x1855007616625152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076809.192979:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616625664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076809.192991:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076809.192995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616625152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076809.193007:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076809.193016:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076809.193035:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3794 00000100:00080000:2.0:1769076809.193037:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076809.193043:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076809.193048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076809.193053:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076809.193057:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076809.193068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616625152:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076809.193085:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616625792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076809.193095:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076809.193098:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076809.193100:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076809.193104:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3794 00000100:00080000:0.0:1769076809.193108:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076809.193112:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076809.193116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616625280:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769076809.193120:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076809.193136:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616625920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076809.193155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076809.193172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076809/real 1769076809] req@ffff995b84643100 x1855007616625280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076809.193185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616625280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076809.193193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076809.193196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076809.193199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616625280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076809.193205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616625408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076809.193219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076809.193226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076809/real 1769076809] req@ffff995b84643480 x1855007616625408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076809.193280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616625408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076809.193295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076809.193300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076809.193304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616625408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076809.193312:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616625536:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076809.193353:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076809.193366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076809/real 1769076809] req@ffff995b84641880 x1855007616625536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076809.193379:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616625536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076809.193388:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076809.193392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076809.193395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616625536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076809.193401:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616625664:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076809.193417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076809.193431:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076809/real 1769076809] req@ffff995b84643b80 x1855007616625664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076809.193447:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616625664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076809.193548:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076809.193555:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076809.193560:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616625664:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076809.193569:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616625792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076809.193595:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076809.193607:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076809/real 1769076809] req@ffff995b84641500 x1855007616625792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076809.193618:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616625792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076809.193625:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076809.193628:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076809.193630:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616625792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076809.193660:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616625920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076809.193688:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076809.193705:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076809/real 1769076809] req@ffff995b84641180 x1855007616625920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076809.193722:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616625920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076809.193734:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076809.193739:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076809.193745:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616625920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076809.896577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076809.896691:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076810.216649:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076810.216650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076810.216673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076810.216732:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076810.920611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076810.920638:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076811.240554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076811.240562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076811.240567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076811.240606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076811.944593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076811.944621:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076812.264533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076812.264545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076812.264580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076812.264607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076812.968592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076812.968694:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076813.288537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076813.288563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076813.288566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076813.288568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076813.992558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076813.992657:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076814.312530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076814.312619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076814.312663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076814.312667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076814.312712:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076814.312718:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076814.312722:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076814.312732:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3800 00000100:00080000:3.0:1769076814.312744:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076814.312753:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076814.312790:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616626048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076814.312815:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076814.312818:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076814.312819:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076814.312823:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3800 00000100:00080000:3.0:1769076814.312827:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076814.312832:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076814.312841:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076814.312856:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616626176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076814.312865:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616626048:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076814.312869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076814.312873:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076814.312876:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076814.312886:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3800 00000100:00080000:3.0:1769076814.312892:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076814.312900:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076814.312907:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076814.312928:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616626304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076814.312943:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076814.312950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076814.312953:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076814.312956:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076814.312961:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3800 00000100:00080000:3.0:1769076814.312969:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076814.312974:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076814.312981:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076814.313002:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616626432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076814.313017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076814/real 1769076814] req@ffff995b8455bb80 x1855007616626048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076814.313048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616626048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076814.313058:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076814.313062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076814.313066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616626048:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076814.313093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616626176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076814.313126:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076814.313127:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076814.313134:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076814.313136:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076814.313141:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3800 00000100:00080000:3.0:1769076814.313146:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076814.313152:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076814.313153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076814/real 1769076814] req@ffff995b8455aa00 x1855007616626176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076814.313176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616626176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076814.313186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076814.313191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076814.313196:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616626176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076814.313205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616626304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076814.313247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076814.313265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076814/real 1769076814] req@ffff995b8455b100 x1855007616626304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076814.313283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616626304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076814.313296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769076814.313301:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769076814.313302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076814.313322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616626304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076814.313326:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616626560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076814.313338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616626432:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076814.313342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076814.313349:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076814.313351:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076814.313373:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076814.313388:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076814/real 1769076814] req@ffff995b8455a680 x1855007616626432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076814.313400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616626432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076814.313458:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076814.313464:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3800 00000100:00080000:0.0:1769076814.313498:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076814.313503:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076814.313506:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616626432:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076814.313513:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076814.313522:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076814.313538:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616626688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076814.313544:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616626560:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076814.313559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076814.313564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076814.313566:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076814.313572:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3800 00000100:00080000:3.0:1769076814.313577:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076814.313583:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769076814.313584:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076814.313589:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076814.313602:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616626816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076814.313638:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076814/real 1769076814] req@ffff995b84558a80 x1855007616626560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076814.313659:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616626560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076814.313672:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076814.313675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076814.313678:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616626560:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076814.313690:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616626688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076814.313709:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076814.313718:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076814/real 1769076814] req@ffff995b84559500 x1855007616626688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076814.313728:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616626688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076814.313735:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076814.313737:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076814.313740:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616626688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076814.313745:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616626816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076814.313757:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076814.313765:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076814/real 1769076814] req@ffff995b84558000 x1855007616626816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076814.313774:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616626816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076814.313781:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076814.313783:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076814.313795:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616626816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076815.016605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076815.016715:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076815.336576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076815.336576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076815.336599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076815.336639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076816.040553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076816.040668:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076816.360551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076816.360560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076816.360564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076816.360592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076817.064617:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076817.064654:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076817.384523:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076817.384534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076817.384552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076817.384615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076818.088568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076818.088598:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076818.408552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076818.408552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076818.408591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076818.408697:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076819.112546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076819.112742:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076819.432599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076819.432642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076819.432657:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076819.432677:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076819.432683:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076819.432690:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076819.432702:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3805 00000100:00080000:2.0:1769076819.432715:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076819.432722:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076819.432773:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076819.432804:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616626944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076819.432861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076819.432868:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076819.432871:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076819.432878:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3805 00000100:00080000:2.0:1769076819.432884:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076819.432891:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076819.432904:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076819.432921:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616627072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076819.432935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616626944:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076819.432943:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076819.432950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076819.432953:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076819.432963:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3805 00000100:00080000:2.0:1769076819.432971:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076819.432979:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076819.432990:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076819.433034:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076819.433038:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616627200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076819.433077:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076819.433081:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076819.433083:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076819.433088:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3805 00000100:00080000:2.0:1769076819.433094:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076819.433100:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076819.433107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076819/real 1769076819] req@ffff995b85ac9880 x1855007616626944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076819.433110:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076819.433132:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616627328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076819.433140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616626944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076819.433208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076819.433209:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076819.433219:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076819.433223:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076819.433224:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076819.433259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616626944:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076819.433260:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3805 00000100:00080000:2.0:1769076819.433270:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076819.433278:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076819.433286:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076819.433301:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616627456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076819.433307:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627072:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076819.433374:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076819.433380:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076819.433382:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076819.433388:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3805 00000100:00080000:2.0:1769076819.433393:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076819.433398:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076819.433404:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076819.433410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076819.433424:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616627584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076819.433432:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076819/real 1769076819] req@ffff995b85acaa00 x1855007616627072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076819.433441:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076819.433452:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076819.433455:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076819.433459:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3805 00000100:00080000:2.0:1769076819.433464:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076819.433468:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076819.433487:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616627072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076819.433497:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076819.433501:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076819.433509:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627072:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769076819.433512:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076819.433519:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076819.433540:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616627712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076819.433553:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076819.433593:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076819/real 1769076819] req@ffff995b85ac9c00 x1855007616627200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076819.433613:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616627200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076819.433640:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076819.433645:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076819.433651:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076819.433660:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627328:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076819.433693:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076819.433722:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076819/real 1769076819] req@ffff995b85acb800 x1855007616627328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076819.433735:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616627328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076819.433755:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076819.433758:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076819.433762:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627328:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076819.433800:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627456:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076819.433822:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076819.433835:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076819/real 1769076819] req@ffff995b85aca300 x1855007616627456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076819.433850:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616627456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076819.433870:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076819.433873:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076819.433878:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076819.433885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616627584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076819.433902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076819.433913:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076819/real 1769076819] req@ffff995b85ac8a80 x1855007616627584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076819.433927:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616627584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076819.433938:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076819.433952:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076819.433956:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616627584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076819.433962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616627712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076819.433980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076819.433991:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076819/real 1769076819] req@ffff995b85acb100 x1855007616627712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076819.434005:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616627712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076819.434016:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076819.434019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076819.434023:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616627712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076820.136584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076820.136610:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076820.456562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076820.456640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076820.456648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076820.456695:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076821.160557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076821.160664:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076821.480555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076821.480555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076821.480592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076821.480597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076822.184528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076822.184572:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076822.504517:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076822.504526:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076822.504527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076822.504531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076823.208542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076823.208705:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076823.528581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076823.528589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076823.528622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076823.528708:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076824.232577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076824.232636:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076824.552566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076824.552595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076824.552629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076824.552629:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076824.552649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076824.552660:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076824.552667:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076824.552681:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3810 00000100:00080000:3.0:1769076824.552696:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076824.552705:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076824.552747:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616627840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076824.552780:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076824.552785:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076824.552787:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076824.552792:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3810 00000100:00080000:3.0:1769076824.552795:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076824.552799:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076824.552809:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076824.552826:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616627968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076824.552845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616627840:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076824.552850:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076824.552863:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076824.552866:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076824.552874:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3810 00000100:00080000:3.0:1769076824.552882:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076824.552890:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076824.552901:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076824.552923:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616628096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076824.552940:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769076824.552949:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076824.552951:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076824.552958:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076824.552968:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3810 00000100:00080000:3.0:1769076824.552978:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076824.552986:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076824.552996:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076824.553016:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076824/real 1769076824] req@ffff995b8455b480 x1855007616627840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076824.553025:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616628224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076824.553066:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076824.553072:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076824.553075:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076824.553083:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3810 00000100:00080000:3.0:1769076824.553092:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076824.553094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616627840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076824.553106:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076824.553122:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769076824.553127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076824.553139:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076824.553146:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616628352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076824.553170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616627840:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076824.553186:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076824.553191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076824.553195:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076824.553202:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3810 00000100:00080000:3.0:1769076824.553207:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076824.553225:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076824.553226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627968:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076824.553278:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076824.553300:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616628480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076824.553314:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076824.553318:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076824.553338:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076824.553348:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3810 00000800:00000100:0.0:1769076824.553353:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076824.553356:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076824.553366:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076824.553374:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076824.553393:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616628608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076824.553400:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076824/real 1769076824] req@ffff995b84559880 x1855007616627968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076824.553438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616627968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076824.553456:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076824.553464:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076824.553513:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616627968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076824.553527:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076824.553592:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076824.553635:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076824/real 1769076824] req@ffff995b8455ad80 x1855007616628096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076824.553673:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616628096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076824.553693:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076824.553700:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076824.553707:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076824.553719:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628224:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076824.553775:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076824.553799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076824/real 1769076824] req@ffff995b8455a300 x1855007616628224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076824.553825:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616628224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076824.553844:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076824.553851:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076824.553859:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628224:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076824.553872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628352:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076824.553912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076824.553937:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076824/real 1769076824] req@ffff995b84559c00 x1855007616628352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076824.553962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616628352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076824.554010:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076824.554017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076824.554024:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076824.554061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616628480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076824.554119:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076824.554144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076824/real 1769076824] req@ffff995b84558700 x1855007616628480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076824.554172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616628480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076824.554191:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076824.554198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076824.554205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616628480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076824.554219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616628608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076824.554306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076824.554330:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076824/real 1769076824] req@ffff995b8455b480 x1855007616628608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076824.554354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616628608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076824.554372:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076824.554377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076824.554385:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616628608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076825.256601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076825.256694:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076825.576580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076825.576600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076825.576618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076825.576645:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076826.280593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076826.280675:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076826.600587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076826.600595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076826.600662:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076826.600673:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076827.304536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076827.304587:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076827.624561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076827.624578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076827.624592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076827.624619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076828.328555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076828.328617:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076828.648563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076828.648570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076828.648605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076828.648615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076829.352525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076829.352624:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076829.672591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076829.672612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769076829.672613:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076829.672618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076829.672626:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076829.672632:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076829.672648:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3815 00000100:00100000:3.0:1769076829.672651:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076829.672667:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076829.672678:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076829.672732:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616628736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076829.672759:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076829.672762:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076829.672764:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076829.672767:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3815 00000100:00080000:1.0:1769076829.672770:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076829.672774:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076829.672783:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076829.672796:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616628864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076829.672803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616628736:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076829.672812:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076829.672816:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076829.672818:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076829.672822:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3815 00000100:00080000:1.0:1769076829.672825:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076829.672828:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076829.672832:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076829.672841:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616628992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076829.672849:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076829.672852:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076829.672854:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076829.672857:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3815 00000100:00080000:1.0:1769076829.672877:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076829.672880:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076829.672884:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769076829.672891:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076829.672896:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616629120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076829.672906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076829.672909:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076829.672910:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076829.672914:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3815 00000100:00080000:1.0:1769076829.672918:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076829.672921:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076829.672924:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076829.672934:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616629248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076829.672941:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076829.672945:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076829/real 1769076829] req@ffff995b868c7800 x1855007616628736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076829.672946:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076829.672951:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076829.672964:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3815 00000100:00080000:1.0:1769076829.672970:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076829.672973:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616628736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076829.672980:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076829.672990:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769076829.672992:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769076829.672997:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076829.673003:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616628736:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076829.673016:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616629376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076829.673030:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628864:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076829.673033:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076829.673039:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076829.673041:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076829.673046:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3815 00000100:00080000:1.0:1769076829.673051:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076829.673055:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076829.673060:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076829.673067:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076829.673072:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076829/real 1769076829] req@ffff995b868c4a80 x1855007616628864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076829.673084:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616629504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076829.673088:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616628864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076829.673117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076829.673125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076829.673130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076829.673136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076829.673176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076829.673184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076829/real 1769076829] req@ffff995b868c6300 x1855007616628992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076829.673203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616628992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076829.673212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076829.673215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076829.673218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616628992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076829.673222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616629120:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076829.673235:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076829.673242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076829/real 1769076829] req@ffff995b868c6d80 x1855007616629120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076829.673251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616629120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076829.673257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076829.673259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076829.673262:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616629120:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076829.673266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616629248:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076829.673277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076829.673283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076829/real 1769076829] req@ffff995b868c5500 x1855007616629248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076829.673291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616629248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076829.673298:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076829.673300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076829.673302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616629248:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076829.673316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616629376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076829.673327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076829.673334:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076829/real 1769076829] req@ffff995b868c4700 x1855007616629376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076829.673342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616629376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076829.673349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076829.673351:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076829.673353:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616629376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076829.673357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616629504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076829.673367:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076829.673374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076829/real 1769076829] req@ffff995b868c7480 x1855007616629504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076829.673383:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616629504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076829.673389:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076829.673391:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076829.673393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616629504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076830.376532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076830.376604:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076830.696536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076830.696536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076830.696563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076830.696582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076831.400515:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076831.400608:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076831.720559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076831.720583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076831.720589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076831.720606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076832.424539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076832.424604:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076832.744567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076832.744589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076832.744590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076832.744607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076833.448552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076833.448618:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076833.768596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076833.768606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076833.768608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076833.768608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076834.472518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076834.472588:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076834.792525:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076834.792542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076834.792548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076834.792552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076834.792565:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076834.792570:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076834.792573:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076834.792581:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3820 00000100:00080000:3.0:1769076834.792592:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076834.792598:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076834.792629:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616629632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076834.792652:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076834.792655:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076834.792656:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076834.792659:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3820 00000100:00080000:3.0:1769076834.792662:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076834.792665:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076834.792674:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076834.792685:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616629632:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076834.792691:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616629760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076834.792705:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076834.792709:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076834.792711:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076834.792716:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3820 00000100:00080000:3.0:1769076834.792720:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076834.792725:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076834.792730:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076834.792746:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076834.792747:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616629888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076834.792757:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076834.792759:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076834.792761:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076834.792764:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3820 00000100:00080000:3.0:1769076834.792767:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076834.792770:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076834.792775:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076834.792786:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076834/real 1769076834] req@ffff995b84558000 x1855007616629632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076834.792786:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616630016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076834.792812:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076834.792824:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616629632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076834.792828:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076834.792833:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076834.792836:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076834.792841:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3820 00000100:00080000:0.0:1769076834.792844:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076834.792851:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616629632:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076834.792851:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076834.792861:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076834.792869:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076834.792877:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616629760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076834.792883:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616630144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076834.792892:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076834.792894:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076834.792896:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076834.792898:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3820 00000100:00080000:3.0:1769076834.792901:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076834.792904:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769076834.792912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076834.792923:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076834.792932:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076834/real 1769076834] req@ffff995b84559500 x1855007616629760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076834.792936:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616630272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076834.792957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076834.792960:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616629760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076834.792961:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076834.792964:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076834.792972:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3820 00000100:00080000:0.0:1769076834.792976:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076834.792977:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076834.792984:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076834.792986:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769076834.792988:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076834.792991:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616629760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076834.792998:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616630400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076834.793001:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616629888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076834.793026:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076834.793035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076834/real 1769076834] req@ffff995b84558a80 x1855007616629888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076834.793043:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616629888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076834.793049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076834.793051:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076834.793053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616629888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076834.793057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630016:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076834.793097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076834.793105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076834/real 1769076834] req@ffff995b8455a680 x1855007616630016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076834.793113:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616630016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076834.793119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076834.793121:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076834.793124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630016:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076834.793135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630144:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076834.793145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076834.793151:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076834/real 1769076834] req@ffff995b8455b100 x1855007616630144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076834.793159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616630144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076834.793165:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076834.793167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076834.793168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630144:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076834.793172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616630272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076834.793181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076834.793187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076834/real 1769076834] req@ffff995b8455aa00 x1855007616630272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076834.793193:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616630272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076834.793199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076834.793201:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076834.793203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616630272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076834.793207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616630400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076834.793215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076834.793225:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076834/real 1769076834] req@ffff995b8455bb80 x1855007616630400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076834.793236:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616630400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076834.793243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076834.793246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076834.793249:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616630400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076835.496584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076835.496691:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076835.816608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076835.816619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076835.816626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076835.816676:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076836.520585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076836.520711:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076836.840573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076836.840591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076836.840613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076836.840626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076837.544524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076837.544588:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076837.864554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076837.864554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076837.864564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076837.864572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076838.568667:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076838.568755:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076838.888561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076838.888587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076838.888625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076838.888628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076839.592620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076839.592734:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076839.912538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076839.912542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076839.912558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076839.912565:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076839.912588:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076839.912601:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3825 00000100:00080000:3.0:1769076839.912617:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076839.912621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076839.912627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076839.912632:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076839.912713:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616630528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076839.912753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076839.912760:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076839.912764:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076839.912772:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3825 00000100:00080000:3.0:1769076839.912780:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076839.912789:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076839.912803:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076839.912829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616630528:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076839.912853:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616630656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076839.912882:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076839.912890:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076839.912895:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076839.912903:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3825 00000100:00080000:3.0:1769076839.912912:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076839.912920:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769076839.912944:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076839.912946:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076839.912964:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616630784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076839.912993:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076839.912997:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076839.913001:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076839.913006:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3825 00000100:00080000:3.0:1769076839.913012:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076839.913017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076839/real 1769076839] req@ffff995b8455bb80 x1855007616630528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076839.913020:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076839.913040:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076839.913052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616630528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076839.913055:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616630912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076839.913080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076839.913082:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076839.913088:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076839.913090:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076839.913091:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076839.913118:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3825 00000100:00100000:1.0:1769076839.913126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616630528:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076839.913127:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076839.913140:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076839.913151:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076839.913168:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616631040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076839.913183:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076839.913189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630656:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076839.913190:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076839.913195:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076839.913207:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3825 00000100:00080000:3.0:1769076839.913212:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076839.913217:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076839.913224:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076839.913270:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616631168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076839.913283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076839.913285:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076839.913292:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076839.913296:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076839.913303:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3825 00000100:00080000:3.0:1769076839.913309:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076839.913314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076839/real 1769076839] req@ffff995b8455aa00 x1855007616630656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076839.913318:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076839.913345:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076839.913362:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616631296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076839.913393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616630656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076839.913423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076839.913427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076839.913431:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076839.913438:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076839.913455:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076839.913463:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076839/real 1769076839] req@ffff995b8455b100 x1855007616630784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076839.913491:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616630784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076839.913499:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076839.913501:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076839.913504:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076839.913508:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630912:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076839.913523:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076839.913530:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076839/real 1769076839] req@ffff995b8455a680 x1855007616630912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076839.913539:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616630912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076839.913546:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076839.913548:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076839.913550:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616630912:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076839.913562:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631040:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076839.913572:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076839.913578:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076839/real 1769076839] req@ffff995b84558a80 x1855007616631040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076839.913586:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616631040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076839.913593:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076839.913595:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076839.913597:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076839.913601:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616631168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076839.913612:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076839.913618:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076839/real 1769076839] req@ffff995b84559500 x1855007616631168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076839.913632:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616631168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076839.913649:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076839.913651:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076839.913653:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616631168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076839.913656:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616631296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076839.913666:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076839.913673:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076839/real 1769076839] req@ffff995b84558000 x1855007616631296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076839.913680:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616631296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076839.913686:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076839.913688:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076839.913690:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616631296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076840.616547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076840.616622:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076840.936700:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076840.936748:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076840.936802:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076840.936802:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076841.640557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076841.640687:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076841.960600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076841.960607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076841.960635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076841.960656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076842.664582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076842.664672:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076842.984612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076842.984670:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076842.984686:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076842.984717:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076843.688562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076843.688683:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076844.008568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076844.008570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076844.008574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076844.008644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076844.712571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076844.712640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076845.032573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076845.032573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076845.032586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769076845.032587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076845.032590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076845.032609:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076845.032627:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3830 00000100:00080000:0.0:1769076845.032645:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076845.032653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076845.032659:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076845.032706:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616631424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076845.032741:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076845.032745:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076845.032747:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076845.032751:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3830 00000100:00080000:0.0:1769076845.032756:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076845.032779:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076845.032792:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076845.032803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616631424:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076845.032816:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616631552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076845.032834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076845.032838:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076845.032842:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076845.032848:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3830 00000100:00080000:0.0:1769076845.032871:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076845.032876:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076845.032883:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076845.032912:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616631680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076845.032939:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076845.032951:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076845.032954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076845.032962:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076845.032968:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3830 00000100:00080000:0.0:1769076845.032974:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076845.032978:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076845.032987:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076845.033004:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616631808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076845.033021:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076845.033024:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076845.033026:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076845.033032:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3830 00000100:00080000:0.0:1769076845.033038:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076845.033042:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076845.033047:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076845.033051:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076845/real 1769076845] req@ffff995b84642a00 x1855007616631424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076845.033061:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616631936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076845.033082:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076845.033086:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076845.033088:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076845.033092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616631424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076845.033098:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3830 00000100:00080000:0.0:1769076845.033109:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076845.033122:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769076845.033124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769076845.033130:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769076845.033132:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076845.033141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616631424:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076845.033159:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616632064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076845.033174:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631552:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076845.033179:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076845.033187:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076845.033191:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076845.033198:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3830 00000800:00000100:1.0:1769076845.033208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076845.033210:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076845.033220:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076845.033225:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076845/real 1769076845] req@ffff995b84643800 x1855007616631552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076845.033228:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076845.033279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616631552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076845.033287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769076845.033298:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616632192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076845.033298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076845.033337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076845.033357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076845.033405:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076845.033455:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076845/real 1769076845] req@ffff995b84640700 x1855007616631680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076845.033464:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616631680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076845.033472:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076845.033475:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076845.033477:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076845.033482:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631808:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076845.033495:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076845.033501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076845/real 1769076845] req@ffff995b84642680 x1855007616631808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076845.033508:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616631808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076845.033514:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076845.033517:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076845.033518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631808:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076845.033522:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631936:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076845.033532:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076845.033539:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076845/real 1769076845] req@ffff995b84641c00 x1855007616631936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076845.033575:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616631936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076845.033583:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076845.033585:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076845.033587:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616631936:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076845.033614:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616632064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076845.033627:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076845.033635:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076845/real 1769076845] req@ffff995b84640000 x1855007616632064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076845.033642:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616632064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076845.033648:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076845.033651:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076845.033653:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616632064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076845.033657:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616632192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076845.033667:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076845.033673:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076845/real 1769076845] req@ffff995b84640e00 x1855007616632192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076845.033680:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616632192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076845.033691:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076845.033693:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076845.033695:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616632192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076845.736580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076845.736702:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076846.056582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076846.056582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076846.056588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076846.056671:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076846.760546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076846.760608:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076847.080628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076847.080629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076847.080635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076847.080709:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076847.784581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076847.784644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076848.104546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076848.104605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076848.104605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076848.104662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076848.808607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076848.808623:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076849.128585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076849.128586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076849.128619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076849.128650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076849.832574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076849.832640:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076850.152578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076850.152602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076850.152612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076850.152640:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076850.152645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076850.152679:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076850.152687:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3835 00000100:00080000:1.0:1769076850.152698:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076850.152704:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076850.152745:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616632320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076850.152749:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076850.152771:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076850.152776:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076850.152778:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076850.152783:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3835 00000100:00080000:1.0:1769076850.152786:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076850.152790:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076850.152799:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076850.152865:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616632448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076850.152874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076850.152877:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076850.152879:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076850.152883:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3835 00000100:00080000:1.0:1769076850.152887:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076850.152891:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076850.152898:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616632320:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076850.152899:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076850.152912:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616632576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076850.152922:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076850.152925:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076850.152926:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076850.152930:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3835 00000100:00080000:1.0:1769076850.152933:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076850.152937:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076850.152941:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076850.152950:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616632704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076850.152958:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076850.152962:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076850.152965:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076850.152969:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3835 00000800:00000100:2.0:1769076850.152974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076850.152977:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076850.152985:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076850.152992:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076850.153009:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616632832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076850.153014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076850/real 1769076850] req@ffff995b868c5c00 x1855007616632320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076850.153025:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076850.153031:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076850.153033:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076850.153038:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3835 00000100:00080000:1.0:1769076850.153042:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076850.153046:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076850.153051:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076850.153052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616632320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076850.153065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076850.153066:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616632960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076850.153075:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076850.153082:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076850.153084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616632320:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076850.153087:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076850.153091:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076850.153098:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3835 00000100:00100000:2.0:1769076850.153098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616632448:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076850.153114:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076850.153118:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076850.153122:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076850.153139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076850.153156:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616633088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076850.153170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076850/real 1769076850] req@ffff995b868c6a00 x1855007616632448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076850.153186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616632448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076850.153214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076850.153219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076850.153224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616632448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076850.153257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616632576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076850.153297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076850.153314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076850/real 1769076850] req@ffff995b868c7100 x1855007616632576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076850.153325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616632576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076850.153334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076850.153338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076850.153342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616632576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076850.153349:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616632704:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076850.153364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076850.153376:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076850/real 1769076850] req@ffff995b868c7480 x1855007616632704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076850.153386:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616632704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076850.153394:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076850.153397:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076850.153399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616632704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076850.153403:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616632832:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076850.153414:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076850.153421:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076850/real 1769076850] req@ffff995b868c4700 x1855007616632832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076850.153435:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616632832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076850.153442:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076850.153444:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076850.153446:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616632832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076850.153450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616632960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076850.153461:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076850.153468:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076850/real 1769076850] req@ffff995b868c5500 x1855007616632960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076850.153479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616632960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076850.153529:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076850.153531:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076850.153534:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616632960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076850.153538:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616633088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076850.153555:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076850.153564:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076850/real 1769076850] req@ffff995b868c6d80 x1855007616633088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076850.153572:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616633088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076850.153583:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076850.153602:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076850.153605:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616633088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076850.856559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076850.856621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076851.176578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076851.176774:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076851.176818:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076851.176819:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076851.880631:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076851.880839:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076852.200631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076852.200645:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076852.200650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076852.200742:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076852.904591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076852.904597:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076853.224628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076853.224631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076853.224634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076853.224699:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076853.928606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076853.928696:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076854.248578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076854.248669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076854.248683:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076854.248709:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076854.952637:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076854.952775:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076855.272601:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076855.272617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076855.272618:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076855.272627:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076855.272628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076855.272644:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3840 00000100:00080000:3.0:1769076855.272660:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076855.272663:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076855.272673:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076855.272675:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076855.272776:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616633216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076855.272828:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076855.272833:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076855.272835:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076855.272841:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3840 00000100:00080000:3.0:1769076855.272848:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076855.272855:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076855.272867:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076855.272892:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616633216:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076855.272908:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616633344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076855.272923:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076855.272927:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076855.272929:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076855.272934:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3840 00000100:00080000:3.0:1769076855.272939:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076855.272943:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076855.272949:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076855.272961:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616633472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076855.272971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769076855.272974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076855.272976:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076855.272981:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076855.272987:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3840 00000100:00080000:3.0:1769076855.272993:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076855.272998:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076855.273003:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076855.273015:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616633600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076855.273025:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076855.273028:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076855.273030:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076855.273034:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3840 00000100:00100000:2.0:1769076855.273038:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076855/real 1769076855] req@ffff995b8455b480 x1855007616633216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076855.273038:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076855.273047:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076855.273051:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076855.273065:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616633728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076855.273092:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076855.273095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076855.273097:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076855.273104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616633216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076855.273116:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3840 00000100:00080000:3.0:1769076855.273122:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076855.273123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076855.273127:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076855.273134:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076855.273135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076855.273143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616633216:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076855.273148:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616633856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076855.273159:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076855.273162:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076855.273164:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076855.273167:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3840 00000100:00080000:3.0:1769076855.273170:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076855.273173:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076855.273177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616633344:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076855.273178:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076855.273196:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616633984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076855.273215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076855.273232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076855/real 1769076855] req@ffff995b84558700 x1855007616633344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076855.273243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616633344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076855.273251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076855.273254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076855.273257:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616633344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076855.273262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616633472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076855.273274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076855.273282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076855/real 1769076855] req@ffff995b84559c00 x1855007616633472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076855.273291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616633472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076855.273299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076855.273301:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076855.273303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616633472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076855.273308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616633600:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076855.273321:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076855.273327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076855/real 1769076855] req@ffff995b8455a300 x1855007616633600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076855.273336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616633600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076855.273343:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076855.273345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076855.273348:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616633600:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076855.273352:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616633728:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076855.273363:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076855.273394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076855/real 1769076855] req@ffff995b8455ad80 x1855007616633728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076855.273404:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616633728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076855.273412:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076855.273414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076855.273417:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616633728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076855.273422:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616633856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076855.273453:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076855.273461:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076855/real 1769076855] req@ffff995b84559880 x1855007616633856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076855.273470:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616633856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076855.273476:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076855.273478:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076855.273480:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616633856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076855.273494:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616633984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076855.273525:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076855.273556:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076855/real 1769076855] req@ffff995b8455b480 x1855007616633984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076855.273568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616633984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076855.273576:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076855.273579:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076855.273582:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616633984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076855.976529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076855.976625:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076856.296640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076856.296653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076856.296676:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076856.296744:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076857.000607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076857.000638:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076857.320590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076857.320602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076857.320627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076857.320630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076858.024574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076858.024728:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076858.344566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076858.344659:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076858.344719:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076858.344740:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076859.048571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076859.048746:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076859.368574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076859.368588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076859.368614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076859.368682:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076860.072592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076860.072731:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076860.392602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076860.392623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076860.392761:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076860.392798:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076860.392809:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076860.392810:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076860.392821:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076860.392836:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3846 00000100:00080000:2.0:1769076860.392851:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076860.392862:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076860.392910:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616634112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076860.392946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076860.392952:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076860.392954:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076860.392960:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3846 00000100:00080000:2.0:1769076860.392966:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076860.392973:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076860.392985:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076860.392999:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616634112:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076860.393007:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616634240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076860.393025:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076860.393030:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076860.393032:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076860.393038:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3846 00000100:00080000:2.0:1769076860.393043:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076860.393047:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076860.393052:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076860.393069:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616634368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076860.393081:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076860.393085:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076860.393088:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076860.393092:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3846 00000100:00080000:2.0:1769076860.393097:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076860.393104:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769076860.393116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769076860.393117:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076860.393135:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616634496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076860.393147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076860.393152:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076860.393155:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076860.393159:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3846 00000100:00080000:2.0:1769076860.393165:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076860.393169:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076860.393176:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076860.393185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076860/real 1769076860] req@ffff995b85ac8380 x1855007616634112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076860.393193:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616634624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076860.393219:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076860.393222:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076860.393224:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076860.393230:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3846 00000100:00080000:2.0:1769076860.393239:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076860.393262:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076860.393268:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076860.393271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616634112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076860.393287:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616634752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076860.393291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076860.393311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076860.393317:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076860.393325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616634112:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076860.393327:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076860.393333:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076860.393344:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3846 00000100:00080000:2.0:1769076860.393349:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076860.393354:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076860.393360:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076860.393370:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616634240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076860.393389:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616634880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076860.393425:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076860.393450:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076860/real 1769076860] req@ffff995b85acbb80 x1855007616634240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076860.393500:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616634240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076860.393511:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076860.393516:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076860.393520:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616634240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076860.393528:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616634368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076860.393552:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076860.393565:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076860/real 1769076860] req@ffff995b85acb100 x1855007616634368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076860.393580:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616634368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076860.393600:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076860.393604:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076860.393608:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616634368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076860.393616:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616634496:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076860.393667:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076860.393683:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076860/real 1769076860] req@ffff995b85ac8a80 x1855007616634496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076860.393706:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616634496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076860.393718:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076860.393722:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076860.393726:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616634496:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076860.393734:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616634624:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076860.393755:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076860.393766:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076860/real 1769076860] req@ffff995b85aca300 x1855007616634624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076860.393780:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616634624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076860.393790:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076860.393794:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076860.393799:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616634624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076860.393806:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616634752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076860.393823:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076860.393833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076860/real 1769076860] req@ffff995b85acb800 x1855007616634752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076860.393847:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616634752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076860.393857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076860.393861:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076860.393864:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616634752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076860.393887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616634880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076860.393906:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076860.393917:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076860/real 1769076860] req@ffff995b85ac9c00 x1855007616634880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076860.393931:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616634880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076860.393941:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076860.393944:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076860.393948:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616634880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076861.096687:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076861.096708:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076861.416697:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076861.416710:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076861.416717:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076861.416759:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076862.120529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076862.120581:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076862.440547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076862.440561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076862.440565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076862.440571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076863.144628:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076863.144693:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076863.464661:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076863.464664:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076863.464743:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076863.464799:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076864.168592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076864.169605:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076864.488626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076864.488682:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076864.488729:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076864.488746:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076865.192570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076865.192589:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076865.512532:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076865.512537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076865.512541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076865.512561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076865.512587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076865.512592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076865.512595:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076865.512604:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3851 00000100:00080000:2.0:1769076865.512613:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076865.512618:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076865.512650:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616635008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076865.512671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076865.512674:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076865.512675:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076865.512678:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3851 00000100:00080000:2.0:1769076865.512681:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076865.512684:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076865.512691:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076865.512699:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616635136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076865.512707:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076865.512712:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769076865.512713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616635008:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076865.512713:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076865.512721:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3851 00000100:00080000:2.0:1769076865.512724:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076865.512726:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076865.512729:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076865.512739:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616635264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076865.512744:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076865.512747:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076865.512748:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076865.512751:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3851 00000100:00080000:2.0:1769076865.512755:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076865.512757:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076865.512759:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076865.512766:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616635392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076865.512772:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076865.512774:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076865.512776:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076865.512778:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3851 00000100:00080000:2.0:1769076865.512780:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076865.512782:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076865.512786:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076865.512793:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616635520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076865.512799:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076865.512803:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076865.512804:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076865.512807:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3851 00000100:00080000:2.0:1769076865.512810:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076865.512815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616635136:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076865.512815:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076865.512823:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076865.512834:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616635648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076865.512840:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076865.512842:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076865.512843:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076865.512846:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3851 00000100:00080000:2.0:1769076865.512850:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076865.512853:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076865.512856:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076865.512857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616635264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076865.512866:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616635776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076865.512882:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616635392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076865.512900:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616635520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076865.512914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616635648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076865.512927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616635776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076866.216617:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076866.216654:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076866.536702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076866.536721:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076866.536790:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076866.536795:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076867.240674:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076867.240730:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076867.560599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076867.560612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076867.560662:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076867.560715:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076868.264586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076868.264603:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076868.584622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076868.584641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076868.584644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076868.584739:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:1.0:1769076868.584879:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076868.584925:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076868.584944:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076868.584983:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076868.584996:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076868.585009:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769076868.585013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076865/real 1769076868] req@ffff995b85ac9880 x1855007616635136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1769076868.585022:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076868.585036:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076868.585059:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769076868.585061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616635136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076868.585075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076868.585081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076868.585086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616635136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076868.585094:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076865/real 1769076868] req@ffff995b85ac9c00 x1855007616635264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076868.585108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616635264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076868.585119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076868.585122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076868.585125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616635264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076868.585131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076865/real 1769076868] req@ffff995b85acb800 x1855007616635392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076868.585142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616635392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076868.585151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076868.585156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076868.585159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616635392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076868.585167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076865/real 1769076868] req@ffff995b85aca300 x1855007616635520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076868.585179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616635520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076868.585188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076868.585192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076868.585195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616635520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076868.585202:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076865/real 1769076868] req@ffff995b85acaa00 x1855007616635008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076868.585214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616635008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076868.585221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076868.585225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076868.585228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616635008:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769076868.585263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076865/real 1769076868] req@ffff995b85ac8a80 x1855007616635648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076868.585276:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616635648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076868.585285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076868.585288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076868.585292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616635648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076868.585299:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076865/real 1769076868] req@ffff995b85acb100 x1855007616635776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076868.585310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616635776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076868.585316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076868.585320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076868.585324:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616635776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076869.288661:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076869.288697:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076869.608613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076869.608621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076869.608691:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076869.608758:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076870.312538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076870.312658:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076870.632615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076870.632646:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076870.632655:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076870.632672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076870.632682:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076870.632692:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076870.632699:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076870.632714:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3856 00000100:00080000:1.0:1769076870.632729:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076870.632740:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076870.632791:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616635904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076870.632834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076870.632850:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076870.632855:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076870.632864:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3856 00000100:00080000:1.0:1769076870.632872:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076870.632879:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076870.632892:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616635904:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076870.632894:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076870.632923:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616636032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076870.632938:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076870.632942:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076870.632945:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076870.632950:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3856 00000100:00080000:1.0:1769076870.632954:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076870.632957:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076870.632963:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076870.632976:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616636160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076870.632986:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076870.632990:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076870.632992:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076870.632995:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3856 00000100:00080000:1.0:1769076870.632999:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076870.633007:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769076870.633013:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769076870.633016:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076870.633038:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616636288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076870.633053:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076870.633058:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076870.633062:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076870.633069:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3856 00000100:00080000:1.0:1769076870.633077:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076870.633082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076870/real 1769076870] req@ffff995b868c6300 x1855007616635904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076870.633103:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076870.633121:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076870.633131:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616635904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076870.633152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076870.633159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076870.633169:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616636416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076870.633174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616635904:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076870.633204:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076870.633210:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076870.633214:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076870.633224:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3856 00000100:00100000:2.0:1769076870.633227:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636032:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076870.633233:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076870.633247:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076870.633258:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076870.633285:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076870.633305:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616636544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076870.633325:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076870/real 1769076870] req@ffff995b868c4a80 x1855007616636032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076870.633328:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076870.633344:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076870.633351:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076870.633363:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3856 00000100:00100000:2.0:1769076870.633368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616636032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076870.633373:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076870.633388:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076870.633403:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769076870.633404:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769076870.633417:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076870.633426:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076870.633435:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616636672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076870.633445:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076870.633578:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076870.633601:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076870/real 1769076870] req@ffff995b868c7800 x1855007616636160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076870.633629:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616636160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076870.633645:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076870.633651:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076870.633656:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076870.633666:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636288:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076870.633714:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076870.633730:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076870/real 1769076870] req@ffff995b868c6d80 x1855007616636288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076870.633747:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616636288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076870.633767:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076870.633772:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076870.633776:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076870.633784:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636416:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076870.633811:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076870.633828:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076870/real 1769076870] req@ffff995b868c5500 x1855007616636416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076870.633860:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616636416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076870.633874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076870.633879:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076870.633884:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076870.633913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616636544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076870.633945:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076870.633964:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076870/real 1769076870] req@ffff995b868c4700 x1855007616636544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076870.633983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616636544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076870.633991:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076870.633994:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076870.633998:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616636544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076870.634005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616636672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076870.634028:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076870.634051:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076870/real 1769076870] req@ffff995b868c7480 x1855007616636672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076870.634073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616636672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076870.634084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076870.634089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076870.634095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616636672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076871.336639:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076871.336668:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076871.656629:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076871.656719:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076871.656771:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076871.656808:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076872.360577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076872.360686:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076872.680544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076872.680565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076872.680581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076872.680611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076873.384644:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076873.384852:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076873.704597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076873.704621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076873.704622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076873.704668:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076874.408671:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076874.408763:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076874.728582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076874.728582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076874.728596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076874.728651:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076875.432597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076875.432701:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076875.752558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076875.752658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076875.752660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076875.752670:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076875.752677:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076875.752690:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076875.752696:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076875.752708:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3861 00000100:00080000:3.0:1769076875.752732:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076875.752746:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076875.752807:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616636800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076875.752848:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076875.752855:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076875.752860:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076875.752871:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3861 00000100:00080000:3.0:1769076875.752880:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076875.752890:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076875.752907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616636800:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769076875.752908:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076875.752957:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616636928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076875.752983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076875.752989:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076875.752993:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076875.752998:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3861 00000100:00080000:3.0:1769076875.753010:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076875.753016:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076875.753025:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076875.753033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076875.753048:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616637056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076875.753070:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076875.753077:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076875.753083:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076875.753094:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3861 00000100:00080000:3.0:1769076875.753106:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076875.753109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076875/real 1769076875] req@ffff995b84558000 x1855007616636800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076875.753127:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076875.753143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616636800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076875.753145:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769076875.753190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076875.753194:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616637184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076875.753200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076875.753213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616636800:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076875.753242:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076875.753250:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769076875.753257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636928:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076875.753258:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076875.753298:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3861 00000100:00080000:3.0:1769076875.753317:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076875.753326:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076875.753338:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769076875.753340:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076875.753414:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616637312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076875.753415:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076875/real 1769076875] req@ffff995b84559500 x1855007616636928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076875.753517:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076875.753547:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076875.753562:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076875.753574:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616636928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076875.753584:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3861 00000100:00080000:2.0:1769076875.753596:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076875.753602:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076875.753606:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076875.753639:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076875.753644:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616636928:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076875.753657:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076875.753662:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076875.753710:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616637440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076875.753731:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076875.753751:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076875/real 1769076875] req@ffff995b84558a80 x1855007616637056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076875.753756:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076875.753791:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076875.753808:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076875.753818:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616637056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076875.753819:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3861 00000100:00080000:3.0:1769076875.753838:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076875.753845:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076875.753852:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076875.753858:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076875.753874:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637056:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769076875.753880:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076875.753885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076875.753922:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616637568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076875.753923:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076875.753944:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076875/real 1769076875] req@ffff995b8455a680 x1855007616637184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076875.753997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616637184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076875.754007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076875.754012:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076875.754020:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076875.754048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637312:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076875.754075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076875.754091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076875/real 1769076875] req@ffff995b8455b100 x1855007616637312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076875.754108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616637312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076875.754121:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076875.754127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076875.754131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076875.754141:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616637440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076875.754168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076875.754180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076875/real 1769076875] req@ffff995b8455aa00 x1855007616637440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076875.754192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616637440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076875.754200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076875.754203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076875.754206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616637440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076875.754212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616637568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076875.754227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076875.754236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076875/real 1769076875] req@ffff995b8455bb80 x1855007616637568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076875.754246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616637568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076875.754253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076875.754256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076875.754258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616637568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076876.456611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076876.456680:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076876.776585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076876.776605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076876.776652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076876.776722:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076877.480598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076877.480688:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076877.800640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076877.800848:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076877.800873:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076877.800909:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076878.504571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076878.504719:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076878.824613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076878.824615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076878.824618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076878.824647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076879.528573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076879.528684:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076879.848549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076879.848619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076879.848621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076879.849535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076880.552590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076880.552704:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076880.872555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076880.872574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076880.872587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076880.872591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076880.872596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076880.872603:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076880.872607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076880.872615:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3866 00000100:00080000:0.0:1769076880.872629:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076880.872639:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076880.872696:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616637696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076880.872726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076880.872732:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076880.872734:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076880.872739:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3866 00000100:00080000:0.0:1769076880.872745:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076880.872750:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076880.872760:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076880.872769:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616637696:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076880.872777:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616637824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076880.872790:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076880.872794:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076880.872796:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076880.872799:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3866 00000100:00080000:0.0:1769076880.872804:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076880.872807:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076880.872812:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076880.872824:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616637952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076880.872833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076880.872850:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076880.872854:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076880.872855:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076880.872862:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3866 00000100:00080000:0.0:1769076880.872867:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076880.872872:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076880.872878:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076880.872890:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616638080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076880.872899:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076880.872903:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076880/real 1769076880] req@ffff995b84640380 x1855007616637696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076880.872904:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076880.872915:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076880.872924:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3866 00000100:00080000:0.0:1769076880.872932:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076880.872939:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076880.872944:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076880.872949:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616637696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076880.872959:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616638208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076880.872977:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076880.872983:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076880.872984:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076880.872986:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076880.872996:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076880.872997:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3866 00000100:00080000:0.0:1769076880.873006:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076880.873009:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616637696:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076880.873014:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076880.873022:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076880.873032:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616638336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076880.873043:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076880.873044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637824:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076880.873051:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076880.873057:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076880.873071:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3866 00000100:00080000:0.0:1769076880.873075:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076880.873078:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076880.873084:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076880.873096:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616638464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076880.873103:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076880.873134:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076880/real 1769076880] req@ffff995b84641f80 x1855007616637824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076880.873181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616637824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076880.873193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076880.873197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076880.873202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076880.873213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076880.873241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076880.873253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076880/real 1769076880] req@ffff995b84640a80 x1855007616637952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076880.873264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616637952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076880.873312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076880.873316:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076880.873320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616637952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076880.873327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638080:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076880.873357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076880.873368:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076880/real 1769076880] req@ffff995b84640e00 x1855007616638080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076880.873388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616638080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076880.873401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076880.873404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076880.873407:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076880.873413:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638208:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076880.873429:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076880.873438:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076880/real 1769076880] req@ffff995b84640000 x1855007616638208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076880.873448:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616638208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076880.873501:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076880.873505:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076880.873509:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638208:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076880.873532:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616638336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076880.873551:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076880.873561:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076880/real 1769076880] req@ffff995b84641c00 x1855007616638336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076880.873572:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616638336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076880.873591:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076880.873593:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076880.873596:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616638336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076880.873601:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616638464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076880.873614:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076880.873622:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076880/real 1769076880] req@ffff995b84642680 x1855007616638464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076880.873632:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616638464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076880.873639:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076880.873651:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076880.873654:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616638464:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769076881.576652:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076881.576694:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076881.896539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076881.896553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076881.896578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076881.896607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076882.600517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076882.600586:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076882.920587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076882.920591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076882.920592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076882.920653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076883.624570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076883.624687:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076883.944607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076883.944650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076883.944669:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076883.944716:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076884.648575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076884.648660:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076884.968601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076884.968614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076884.968718:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076884.968738:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076885.672578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076885.672654:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076885.992551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076885.992553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076885.992562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076885.992573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076885.992581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769076885.992583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076885.992587:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076885.992600:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3871 00000100:00080000:0.0:1769076885.992612:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076885.992632:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076885.992666:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616638592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076885.992700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076885.992703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076885.992705:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076885.992708:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3871 00000100:00080000:0.0:1769076885.992711:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076885.992715:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076885.992723:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076885.992732:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616638720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076885.992736:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616638592:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076885.992749:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076885.992752:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076885.992754:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076885.992757:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3871 00000100:00080000:0.0:1769076885.992760:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076885.992762:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076885.992765:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076885.992775:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616638848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076885.992782:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076885.992784:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076885.992785:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076885.992788:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3871 00000100:00080000:0.0:1769076885.992792:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076885.992794:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076885.992797:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076885.992816:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616638976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076885.992817:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076885.992825:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076885.992828:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076885.992846:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076885.992852:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3871 00000100:00080000:0.0:1769076885.992866:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076885.992870:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076885.992875:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076885.992900:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616639104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076885.992908:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076885/real 1769076885] req@ffff995b84640700 x1855007616638592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076885.992917:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076885.992929:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076885.992934:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076885.992945:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3871 00000100:00080000:0.0:1769076885.992950:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076885.992953:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076885.992968:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616638592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076885.992970:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076885.992991:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616639232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076885.992995:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076885.993010:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076885.993012:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076885.993021:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076885.993024:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616638592:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076885.993026:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076885.993040:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3871 00000100:00080000:0.0:1769076885.993045:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076885.993049:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076885.993054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076885.993068:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616639360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076885.993078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076885.993144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076885.993155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076885/real 1769076885] req@ffff995b84643800 x1855007616638720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076885.993164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616638720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076885.993170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076885.993172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076885.993175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076885.993182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076885.993218:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076885.993265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076885/real 1769076885] req@ffff995b84642a00 x1855007616638848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076885.993290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616638848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076885.993306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076885.993312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076885.993318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076885.993330:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076885.993393:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076885.993419:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076885/real 1769076885] req@ffff995b84641180 x1855007616638976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076885.993435:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616638976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076885.993445:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076885.993449:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076885.993452:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616638976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076885.993459:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616639104:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076885.993499:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076885.993513:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076885/real 1769076885] req@ffff995b84641500 x1855007616639104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076885.993529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616639104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076885.993538:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076885.993542:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076885.993546:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616639104:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076885.993552:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616639232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076885.993570:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076885.993583:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076885/real 1769076885] req@ffff995b84643b80 x1855007616639232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076885.993591:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616639232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076885.993608:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076885.993612:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076885.993615:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616639232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076885.993641:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616639360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076885.993665:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076885.993681:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076885/real 1769076885] req@ffff995b84641880 x1855007616639360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076885.993699:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616639360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076885.993713:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076885.993716:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076885.993718:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616639360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076886.696560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076886.696610:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076887.016581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076887.016592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076887.016595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076887.016622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076887.720523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076887.720703:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076888.040594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076888.040594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076888.040645:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076888.040645:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076888.744600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076888.744675:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076889.064536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076889.064561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076889.064562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076889.064564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076889.768596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076889.768770:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076890.088621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076890.088627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076890.088632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076890.088685:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076890.792636:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076890.792724:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076891.112584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076891.112589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076891.112616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076891.112642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076891.112662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076891.112669:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076891.112674:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076891.112684:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3876 00000100:00080000:2.0:1769076891.112698:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076891.112707:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076891.112762:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616639488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076891.112793:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076891.112798:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076891.112800:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076891.112804:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3876 00000100:00080000:2.0:1769076891.112808:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076891.112813:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076891.112822:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076891.112836:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616639616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076891.112845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616639488:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076891.112864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076891.112881:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076891.112883:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076891.112888:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3876 00000100:00080000:2.0:1769076891.112892:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076891.112896:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076891.112901:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076891.112913:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616639744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076891.112924:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076891.112928:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076891.112930:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076891.112936:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076891.112941:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3876 00000100:00080000:2.0:1769076891.112947:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076891.112952:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076891.112958:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076891.112970:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616639872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076891.112981:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076891.112985:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076891.112991:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076891/real 1769076891] req@ffff995b85acb100 x1855007616639488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076891.112991:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076891.113001:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3876 00000100:00080000:2.0:1769076891.113007:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076891.113015:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076891.113023:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076891.113032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616639488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076891.113037:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616640000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076891.113071:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076891.113072:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076891.113082:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076891.113086:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076891.113088:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076891.113098:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616639488:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076891.113099:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3876 00000100:00080000:2.0:1769076891.113107:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076891.113113:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076891.113179:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076891.113193:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616639616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076891.113201:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616640128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076891.113228:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076891.113233:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076891.113236:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076891.113244:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076891.113251:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3876 00000100:00080000:2.0:1769076891.113261:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076891.113268:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076891.113275:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076891.113276:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076891/real 1769076891] req@ffff995b85ac8a80 x1855007616639616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076891.113294:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616640256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076891.113303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616639616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076891.113399:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076891.113403:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076891.113407:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616639616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076891.113416:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616639744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076891.113440:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076891.113450:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076891/real 1769076891] req@ffff995b85acaa00 x1855007616639744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076891.113462:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616639744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076891.113472:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076891.113475:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076891.113478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616639744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076891.113508:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616639872:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076891.113529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076891.113540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076891/real 1769076891] req@ffff995b85aca300 x1855007616639872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076891.113552:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616639872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076891.113561:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076891.113564:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076891.113568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616639872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076891.113574:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640000:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076891.113588:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076891.113597:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076891/real 1769076891] req@ffff995b85acb800 x1855007616640000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076891.113608:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616640000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076891.113617:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076891.113620:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076891.113622:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076891.113639:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616640128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076891.113654:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076891.113663:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076891/real 1769076891] req@ffff995b85ac9c00 x1855007616640128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076891.113675:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616640128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076891.113683:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076891.113686:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076891.113689:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616640128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076891.113694:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616640256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076891.113708:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076891.113718:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076891/real 1769076891] req@ffff995b85ac9880 x1855007616640256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076891.113729:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616640256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076891.113737:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076891.113740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076891.113743:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616640256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076891.816624:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076891.816719:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076892.136660:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076892.136684:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076892.136690:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076892.136736:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076892.840585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076892.840693:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076893.160616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076893.160647:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076893.160665:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076893.160736:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076893.864529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076893.864679:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076894.184570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076894.184584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076894.184600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076894.184603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076894.888574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076894.888640:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076895.208601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076895.208634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076895.208634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076895.208657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076895.912599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076895.912681:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076896.232620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076896.232620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076896.232661:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076896.232690:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076896.232715:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076896.232717:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076896.232728:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076896.232742:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3881 00000100:00080000:0.0:1769076896.232759:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076896.232769:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076896.232827:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616640384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076896.232872:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076896.232878:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076896.232880:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076896.232886:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3881 00000100:00080000:0.0:1769076896.232890:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076896.232895:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076896.232907:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076896.232923:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616640512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076896.232937:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076896.232941:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076896.232944:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076896.232951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616640384:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076896.232952:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3881 00000100:00080000:0.0:1769076896.232960:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076896.232971:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076896.232977:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076896.233014:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616640640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076896.233023:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076896.233026:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076896.233047:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076896.233054:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3881 00000100:00080000:0.0:1769076896.233059:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076896.233063:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076896.233069:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076896.233085:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616640768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076896.233112:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076896.233116:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076896.233118:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076896.233123:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3881 00000100:00080000:0.0:1769076896.233136:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076896.233140:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076896.233146:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769076896.233154:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076896.233166:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616640896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076896.233185:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076896.233191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076896.233194:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076896.233201:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3881 00000100:00080000:0.0:1769076896.233209:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076896.233216:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076896.233223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076896/real 1769076896] req@ffff995b84643480 x1855007616640384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076896.233229:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076896.233259:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616641024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076896.233266:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616640384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076896.233319:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076896.233325:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076896.233325:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076896.233331:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076896.233341:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3881 00000100:00080000:1.0:1769076896.233341:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076896.233348:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076896.233359:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076896.233360:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616640384:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076896.233365:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076896.233385:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616641152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076896.233414:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076896.233471:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076896.233527:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076896/real 1769076896] req@ffff995b84643100 x1855007616640512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076896.233549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616640512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076896.233563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076896.233569:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076896.233574:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076896.233602:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076896.233634:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076896.233651:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076896/real 1769076896] req@ffff995b84642d80 x1855007616640640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076896.233673:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616640640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076896.233691:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076896.233697:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076896.233703:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076896.233712:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640768:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076896.233737:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076896.233751:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076896/real 1769076896] req@ffff995b84641880 x1855007616640768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076896.233782:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616640768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076896.233797:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076896.233802:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076896.233808:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076896.233816:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076896.233837:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076896.233850:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076896/real 1769076896] req@ffff995b84643b80 x1855007616640896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076896.233869:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616640896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076896.233885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076896.233889:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076896.233894:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616640896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076896.233903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616641024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076896.233925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076896.233939:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076896/real 1769076896] req@ffff995b84641500 x1855007616641024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076896.233958:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616641024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076896.233997:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076896.234002:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076896.234007:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616641024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076896.234015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616641152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076896.234076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076896.234092:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076896/real 1769076896] req@ffff995b84641180 x1855007616641152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076896.234109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616641152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076896.234122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076896.234127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076896.234132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616641152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076896.936561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076896.936610:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076897.256598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076897.256617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076897.256665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076897.256674:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076897.960630:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076897.960640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076898.280590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076898.280597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076898.280688:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076898.280700:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076898.984584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076898.984619:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076899.304605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076899.304612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076899.304646:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076899.304738:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076900.008584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076900.008654:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076900.328606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076900.328619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076900.328704:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076900.328712:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076901.032567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076901.032644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076901.352612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076901.352627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076901.352708:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076901.352716:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076901.352739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076901.352748:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076901.352754:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076901.352767:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3887 00000100:00080000:0.0:1769076901.352778:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076901.352786:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076901.352826:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616641280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076901.352849:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076901.352852:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076901.352854:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076901.352857:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3887 00000100:00080000:0.0:1769076901.352860:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076901.352865:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076901.352872:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076901.352893:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616641408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076901.352903:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076901.352906:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076901.352908:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076901.352919:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3887 00000100:00100000:1.0:1769076901.352920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616641280:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076901.352935:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076901.352942:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076901.352982:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076901.352998:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616641536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076901.353006:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076901.353008:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076901.353010:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076901.353013:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3887 00000100:00080000:0.0:1769076901.353017:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076901.353020:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076901.353024:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076901.353033:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616641664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076901.353040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076901.353042:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076901.353043:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076901.353045:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3887 00000100:00080000:0.0:1769076901.353053:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076901.353056:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076901.353059:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076901.353071:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616641792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076901.353072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076901.353085:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076901.353094:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076901.353097:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076901.353112:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3887 00000100:00080000:0.0:1769076901.353120:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076901.353124:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076901.353127:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076901.353137:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616641920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076901.353152:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076901.353153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076901/real 1769076901] req@ffff995b84642a00 x1855007616641280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076901.353161:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076901.353164:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076901.353181:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3887 00000100:00080000:0.0:1769076901.353191:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076901.353206:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076901.353210:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076901.353226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616641280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076901.353227:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616642048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076901.353309:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076901.353314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076901.353320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616641280:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076901.353350:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616641408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076901.353370:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076901.353380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076901/real 1769076901] req@ffff995b84643800 x1855007616641408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076901.353390:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616641408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076901.353397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076901.353399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076901.353402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616641408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076901.353406:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616641536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076901.353417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076901.353424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076901/real 1769076901] req@ffff995b84640700 x1855007616641536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076901.353432:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616641536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076901.353446:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076901.353448:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076901.353451:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616641536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076901.353455:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616641664:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076901.353467:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076901.353473:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076901/real 1769076901] req@ffff995b84642680 x1855007616641664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076901.353481:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616641664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076901.353488:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076901.353490:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076901.353492:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616641664:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076901.353520:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616641792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076901.353558:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076901.353573:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076901/real 1769076901] req@ffff995b84641c00 x1855007616641792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076901.353589:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616641792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076901.353602:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076901.353607:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076901.353612:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616641792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076901.353621:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616641920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076901.353644:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076901.353654:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076901/real 1769076901] req@ffff995b84640000 x1855007616641920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076901.353664:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616641920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076901.353672:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076901.353674:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076901.353677:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616641920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076901.353682:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616642048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076901.353695:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076901.353702:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076901/real 1769076901] req@ffff995b84640e00 x1855007616642048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076901.353711:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616642048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076901.353718:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076901.353720:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076901.353722:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616642048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076902.056563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076902.056702:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076902.376600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076902.376600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076902.376606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076902.376635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076903.080574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076903.080638:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076903.400602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076903.400661:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076903.400697:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076903.400725:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076904.104567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076904.104656:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076904.424539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076904.424545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076904.424591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076904.424610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076905.128519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076905.128580:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076905.448550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076905.448567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076905.448589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076905.448591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076906.152564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076906.152624:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076906.472576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076906.472586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769076906.472586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076906.472591:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076906.472601:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3892 00000100:00100000:1.0:1769076906.472607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076906.472613:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076906.472621:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076906.472643:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076906.472646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076906.472657:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616642176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076906.472688:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076906.472695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076906.472698:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076906.472704:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3892 00000100:00080000:0.0:1769076906.472710:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076906.472716:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076906.472724:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616642176:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076906.472728:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076906.472753:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616642304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076906.472764:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076906.472767:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076906.472770:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076906.472773:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3892 00000100:00080000:0.0:1769076906.472776:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076906.472779:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076906.472786:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076906.472794:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616642432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076906.472805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076906.472809:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076906.472812:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769076906.472819:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076906.472825:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3892 00000100:00080000:0.0:1769076906.472835:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076906.472840:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076906.472849:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076906.472892:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616642560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076906.472900:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076906/real 1769076906] req@ffff995b84640a80 x1855007616642176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076906.472905:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076906.472923:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076906.472932:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076906.472941:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3892 00000100:00080000:0.0:1769076906.472946:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076906.472952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616642176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076906.472955:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076906.472978:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769076906.472978:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076906.472987:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076906.472992:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616642176:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076906.473004:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616642688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076906.473017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076906.473019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076906.473021:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076906.473024:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3892 00000100:00080000:0.0:1769076906.473028:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076906.473032:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076906.473036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616642304:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769076906.473040:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076906.473054:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616642816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076906.473062:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076906.473070:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1769076906.473070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076906.473077:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076906.473084:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3892 00000100:00080000:0.0:1769076906.473089:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076906.473096:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076906/real 1769076906] req@ffff995b84641f80 x1855007616642304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076906.473097:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076906.473106:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076906.473123:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616642944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076906.473127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616642304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076906.473176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076906.473183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076906.473188:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616642304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076906.473195:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616642432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076906.473214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076906.473222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076906/real 1769076906] req@ffff995b84640380 x1855007616642432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076906.473232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616642432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076906.473241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076906.473245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076906.473249:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616642432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076906.473257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616642560:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076906.473283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076906.473298:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076906/real 1769076906] req@ffff995b84640e00 x1855007616642560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076906.473315:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616642560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076906.473326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076906.473328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076906.473332:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616642560:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076906.473338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616642688:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076906.473358:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076906.473372:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076906/real 1769076906] req@ffff995b84640000 x1855007616642688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076906.473387:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616642688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076906.473398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076906.473403:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076906.473407:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616642688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076906.473431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616642816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076906.473478:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076906.473489:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076906/real 1769076906] req@ffff995b84641c00 x1855007616642816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076906.473500:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616642816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076906.473507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076906.473510:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076906.473512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616642816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076906.473517:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616642944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076906.473532:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076906.473540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076906/real 1769076906] req@ffff995b84642680 x1855007616642944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076906.473549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616642944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076906.473555:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076906.473557:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076906.473560:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616642944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076907.176595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076907.176757:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076907.496583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076907.496585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076907.496615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076907.496620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076908.200625:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076908.200664:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076908.520544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076908.520544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076908.520597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076908.520605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076909.224628:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076909.224749:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076909.544580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076909.544610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076909.544638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076909.544859:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076910.248548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076910.248645:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076910.568605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076910.568608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076910.568610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076910.568683:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076911.272588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076911.272735:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076911.592647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076911.592670:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076911.592712:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076911.592761:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076911.592800:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076911.592813:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076911.592821:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076911.592839:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3897 00000100:00080000:1.0:1769076911.592858:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076911.592871:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076911.592927:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616643072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076911.592977:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076911.592982:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076911.592985:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076911.592992:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3897 00000100:00080000:1.0:1769076911.592997:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076911.593004:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076911.593019:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076911.593043:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616643200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076911.593068:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076911.593070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616643072:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076911.593082:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076911.593092:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076911.593103:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3897 00000100:00080000:1.0:1769076911.593111:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076911.593120:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076911.593132:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076911.593168:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616643328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076911.593189:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769076911.593194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076911.593204:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076911.593211:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076911.593223:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3897 00000100:00080000:1.0:1769076911.593232:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076911.593240:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076911.593261:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076911.593270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076911/real 1769076911] req@ffff995b868c7100 x1855007616643072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076911.593286:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616643456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076911.593303:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076911.593309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616643072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076911.593310:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076911.593315:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076911.593328:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3897 00000100:00080000:1.0:1769076911.593339:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076911.593351:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769076911.593353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769076911.593358:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769076911.593367:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076911.593376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616643072:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076911.593381:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616643584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076911.593396:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076911.593401:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076911.593403:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076911.593408:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3897 00000100:00080000:1.0:1769076911.593413:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076911.593418:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076911.593427:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616643200:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769076911.593440:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076911.593500:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616643712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076911.593514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076911.593525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076911.593528:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076911.593547:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3897 00000100:00080000:1.0:1769076911.593554:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076911.593559:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076911.593567:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076911.593580:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076911.593588:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616643840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076911.593615:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076911/real 1769076911] req@ffff995b868c6a00 x1855007616643200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076911.593644:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616643200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076911.593656:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076911.593661:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076911.593666:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616643200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076911.593675:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616643328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076911.593697:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076911.593709:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076911/real 1769076911] req@ffff995b868c5c00 x1855007616643328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076911.593724:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616643328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076911.593736:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076911.593740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076911.593743:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616643328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076911.593751:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616643456:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076911.593769:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076911.593822:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076911/real 1769076911] req@ffff995b868c7480 x1855007616643456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076911.593847:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616643456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076911.593859:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076911.593863:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076911.593867:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616643456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076911.593875:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616643584:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076911.593899:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076911.593911:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076911/real 1769076911] req@ffff995b868c4700 x1855007616643584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076911.593925:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616643584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076911.593950:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076911.593955:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076911.593962:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616643584:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076911.593996:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616643712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076911.594040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076911.594065:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076911/real 1769076911] req@ffff995b868c5500 x1855007616643712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076911.594085:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616643712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076911.594104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076911.594108:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076911.594112:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616643712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076911.594120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616643840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076911.594146:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076911.594160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076911/real 1769076911] req@ffff995b868c6d80 x1855007616643840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076911.594174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616643840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076911.594190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076911.594194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076911.594197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616643840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076912.296601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076912.296757:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076912.616565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076912.616586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076912.616595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076912.616655:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076913.320626:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076913.320701:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076913.640555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076913.640576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076913.640579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076913.640590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076914.344518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076914.344611:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076914.664545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076914.664567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076914.664573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076914.664624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076915.368574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076915.368616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076915.688572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076915.688579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076915.688603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076915.688622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076916.392539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076916.392662:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076916.712565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076916.712573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076916.712611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076916.712642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769076916.712667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076916.712674:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076916.712680:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076916.712700:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3902 00000100:00080000:2.0:1769076916.712711:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076916.712716:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076916.712762:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616643968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076916.712786:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076916.712789:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076916.712790:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076916.712794:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3902 00000100:00080000:2.0:1769076916.712797:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076916.712801:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076916.712809:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076916.712830:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616644096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076916.712838:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076916.712846:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076916.712849:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076916.712854:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616643968:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076916.712855:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3902 00000100:00080000:2.0:1769076916.712861:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076916.712866:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076916.712870:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076916.712883:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616644224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076916.712897:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076916.712899:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076916.712901:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076916.712904:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3902 00000100:00080000:2.0:1769076916.712909:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076916.712917:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076916.712921:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769076916.712923:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076916.712935:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616644352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076916.712945:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076916.712947:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076916.712948:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076916.712951:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3902 00000100:00080000:2.0:1769076916.712955:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076916.712960:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076916.712967:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076916.712969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076916/real 1769076916] req@ffff995b85acbb80 x1855007616643968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076916.712984:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616644480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076916.713003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076916.713013:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076916.713017:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076916.713021:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3902 00000100:00080000:2.0:1769076916.713026:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076916.713033:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076916.713033:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616643968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076916.713042:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076916.713049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076916.713057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076916.713059:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616644608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076916.713064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616643968:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076916.713071:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076916.713086:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076916.713087:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076916.713089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644096:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769076916.713093:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3902 00000100:00080000:2.0:1769076916.713098:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076916.713101:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076916.713105:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076916.713117:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616644736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076916.713125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076916.713153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076916/real 1769076916] req@ffff995b85ac8380 x1855007616644096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076916.713164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616644096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076916.713171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076916.713174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076916.713176:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076916.713196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076916.713209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076916.713216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076916/real 1769076916] req@ffff995b85ac9880 x1855007616644224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076916.713225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616644224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076916.713237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076916.713240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076916.713242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076916.713247:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644352:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076916.713258:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076916.713264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076916/real 1769076916] req@ffff995b85ac9c00 x1855007616644352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076916.713273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616644352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076916.713279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076916.713281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076916.713283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076916.713288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644480:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076916.713298:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076916.713304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076916/real 1769076916] req@ffff995b85acb800 x1855007616644480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076916.713312:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616644480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076916.713318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076916.713320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076916.713323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076916.713328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616644608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076916.713338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076916.713365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076916/real 1769076916] req@ffff995b85aca300 x1855007616644608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076916.713378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616644608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076916.713388:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076916.713391:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076916.713395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616644608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076916.713402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616644736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076916.713437:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076916.713476:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076916/real 1769076916] req@ffff995b85acaa00 x1855007616644736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076916.713495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616644736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076916.713516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076916.713520:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076916.713525:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616644736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076917.416553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076917.416655:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076917.736542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076917.736557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076917.736586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076917.736609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076918.440571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076918.440722:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076918.760593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076918.760593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076918.760603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076918.760630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076919.464633:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076919.464779:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076919.784620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076919.784623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076919.784662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076919.784702:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076920.488559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076920.488685:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076920.808605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076920.808684:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076920.808691:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076920.808721:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076921.512557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076921.512646:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076921.832560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769076921.832588:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076921.832596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076921.832619:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076921.832633:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769076921.832633:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3907 00000100:00100000:1.0:1769076921.832657:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076921.832668:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076921.832679:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076921.832687:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076921.832726:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616644864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076921.832753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076921.832757:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076921.832758:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076921.832765:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3907 00000100:00080000:2.0:1769076921.832768:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076921.832772:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076921.832776:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616644864:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769076921.832784:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076921.832799:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616644992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076921.832813:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076921.832817:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076921.832818:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076921.832821:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3907 00000100:00080000:2.0:1769076921.832825:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076921.832829:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076921.832834:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076921.832844:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616645120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076921.832852:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076921.832855:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076921.832858:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076921.832864:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3907 00000100:00080000:2.0:1769076921.832872:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076921.832876:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076921.832882:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076921.832895:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076921.832918:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616645248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076921.832953:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076921.832958:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076921.832962:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076921.832967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076921/real 1769076921] req@ffff995b85ac8a80 x1855007616644864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076921.832971:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3907 00000100:00080000:2.0:1769076921.832981:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076921.832991:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076921.833002:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076921.833011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616644864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076921.833023:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616645376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076921.833035:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076921.833042:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076921.833046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769076921.833053:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769076921.833060:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616644864:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769076921.833060:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076921.833090:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3907 00000100:00080000:2.0:1769076921.833096:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076921.833104:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076921.833114:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076921.833118:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076921.833138:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616645504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076921.833153:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769076921.833155:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076921.833160:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076921.833162:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076921.833167:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3907 00000100:00100000:1.0:1769076921.833171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076921/real 1769076921] req@ffff995b85acb100 x1855007616644992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076921.833173:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076921.833189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616644992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076921.833191:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769076921.833199:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769076921.833202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076921.833207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076921.833210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616644992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076921.833233:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616645632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076921.833234:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616645120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076921.833259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076921.833269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076921/real 1769076921] req@ffff995b85acaa00 x1855007616645120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076921.833279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616645120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076921.833293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076921.833296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076921.833298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616645120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076921.833303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616645248:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076921.833316:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076921.833322:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076921/real 1769076921] req@ffff995b85aca300 x1855007616645248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076921.833330:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616645248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076921.833354:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076921.833357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076921.833359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616645248:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076921.833365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616645376:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076921.833390:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076921.833405:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076921/real 1769076921] req@ffff995b85acb800 x1855007616645376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076921.833437:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616645376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076921.833450:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076921.833455:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076921.833461:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616645376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076921.833491:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616645504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076921.833586:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076921.833619:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076921/real 1769076921] req@ffff995b85ac9c00 x1855007616645504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076921.833649:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616645504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076921.833657:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076921.833661:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076921.833664:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616645504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076921.833670:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616645632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076921.833690:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076921.833701:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076921/real 1769076921] req@ffff995b85ac9880 x1855007616645632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076921.833714:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616645632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076921.833723:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076921.833727:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076921.833730:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616645632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076922.536529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076922.536619:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076922.856571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076922.856668:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076922.856715:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076922.856716:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076923.560528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076923.560660:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076923.880659:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076923.880661:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076923.880685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076923.880706:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076924.584595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076924.584662:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076924.904640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076924.904640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076924.904650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076924.904650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076925.608562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076925.608627:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076925.928567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076925.928585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076925.928596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076925.928644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076926.632568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076926.632702:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076926.952573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076926.952583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076926.952585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076926.952622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076926.952628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076926.952631:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076926.952640:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3912 00000100:00080000:3.0:1769076926.952649:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076926.952655:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076926.952674:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076926.952690:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616645760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076926.952739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076926.952743:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076926.952746:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076926.952750:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3912 00000100:00080000:3.0:1769076926.952753:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076926.952758:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076926.952768:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076926.952779:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616645888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076926.952793:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616645760:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076926.952795:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076926.952812:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076926.952817:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076926.952824:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3912 00000100:00080000:3.0:1769076926.952829:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076926.952835:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076926.952845:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076926.952863:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616646016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076926.952877:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076926.952881:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076926.952891:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076926.952896:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3912 00000100:00080000:3.0:1769076926.952901:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076926.952905:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076926.952910:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769076926.952913:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076926.952944:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616646144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076926.952954:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076926.952957:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076926.952958:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076926.952962:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3912 00000100:00080000:3.0:1769076926.952967:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076926.952969:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076926.952975:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076926.952988:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076926/real 1769076926] req@ffff995b8455bb80 x1855007616645760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076926.952989:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616646272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076926.953031:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076926.953035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616645760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076926.953042:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076926.953062:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076926.953071:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3912 00000100:00080000:1.0:1769076926.953075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076926.953077:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076926.953095:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076926.953096:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076926.953113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616645760:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769076926.953114:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076926.953140:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616646400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076926.953167:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076926.953169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616645888:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076926.953182:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076926.953185:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076926.953191:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3912 00000100:00080000:3.0:1769076926.953195:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076926.953199:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076926.953204:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076926.953212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076926.953224:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616646528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076926.953232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076926/real 1769076926] req@ffff995b8455aa00 x1855007616645888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076926.953277:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616645888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076926.953294:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076926.953297:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076926.953302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616645888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076926.953308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076926.953324:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076926.953346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076926/real 1769076926] req@ffff995b8455b100 x1855007616646016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076926.953378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616646016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076926.953392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076926.953396:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076926.953402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076926.953411:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646144:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076926.953454:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076926.953470:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076926/real 1769076926] req@ffff995b8455a680 x1855007616646144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076926.953487:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616646144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076926.953500:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076926.953504:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076926.953508:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646144:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076926.953581:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769076926.953611:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076926.953630:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076926/real 1769076926] req@ffff995b84558a80 x1855007616646272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076926.953648:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616646272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076926.953663:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076926.953668:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076926.953673:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769076926.953698:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616646400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076926.953731:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076926.953742:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076926/real 1769076926] req@ffff995b84559500 x1855007616646400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076926.953753:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616646400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076926.953761:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076926.953765:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076926.953767:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616646400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076926.953772:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616646528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769076926.953786:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076926.953795:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076926/real 1769076926] req@ffff995b84558000 x1855007616646528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076926.953821:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616646528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076926.953828:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076926.953831:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076926.953833:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616646528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076927.656570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076927.656667:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076927.976556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076927.976608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076927.976619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076927.976691:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076928.680571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076928.680682:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076929.000592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076929.000599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076929.000637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076929.000670:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076929.704577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076929.704680:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076930.024611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076930.024639:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076930.024659:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076930.024683:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076930.728617:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076930.728715:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076931.048594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076931.048636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076931.048637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076931.048854:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076931.752559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076931.752638:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076932.072544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076932.072570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076932.072578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076932.072583:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076932.072596:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3917 00000100:00080000:3.0:1769076932.072608:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076932.072618:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076932.072619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076932.072646:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076932.072652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076932.072660:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616646656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076932.072696:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076932.072701:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076932.072704:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076932.072710:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3917 00000100:00080000:3.0:1769076932.072714:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076932.072719:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076932.072725:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616646656:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769076932.072729:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076932.072752:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616646784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076932.072761:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076932.072764:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076932.072765:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076932.072768:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3917 00000100:00080000:3.0:1769076932.072771:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076932.072774:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076932.072778:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076932.072787:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616646912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076932.072794:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076932.072796:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076932.072798:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076932.072800:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3917 00000100:00080000:3.0:1769076932.072803:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076932.072806:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076932.072809:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076932.072818:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076932.072820:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616647040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076932.072833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076932.072835:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076932.072837:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076932.072839:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3917 00000100:00080000:3.0:1769076932.072843:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076932.072845:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076932.072849:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076932.072854:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076932.072860:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616647168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076932.072869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076932.072872:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076932.072874:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076932.072878:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3917 00000100:00100000:0.0:1769076932.072881:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647040:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076932.072883:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076932.072888:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076932.072892:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076932.072902:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616647296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076932.072904:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647168:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769076932.072917:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076932.072920:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076932.072922:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076932.072927:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3917 00000100:00100000:0.0:1769076932.072929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616647296:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076932.072931:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076932.072937:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076932.072942:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076932.072955:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616647424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076932.072985:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616647424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076932.776513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076932.776583:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076933.096587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076933.096644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076933.096726:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076933.096774:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076933.800545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076933.800595:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076934.120550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076934.120552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076934.120563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076934.120608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076934.824543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076934.824647:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076935.144564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076935.144573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076935.144588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076935.144635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:1.0:1769076935.144839:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769076935.144893:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076935.144915:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076935.144963:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076935.144979:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076935.144990:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076935.145003:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076935.145017:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769076935.145032:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769076935.145047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076932/real 1769076935] req@ffff995b84558700 x1855007616647296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076935.145108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616647296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076935.145125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076935.145133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076935.145140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616647296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076935.145151:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076932/real 1769076935] req@ffff995b8455a300 x1855007616647040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076935.145169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616647040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076935.145183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076935.145189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076935.145194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076935.145203:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076932/real 1769076935] req@ffff995b84559c00 x1855007616647168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076935.145220:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616647168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076935.145232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076935.145236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076935.145240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076935.145248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076932/real 1769076935] req@ffff995b84559880 x1855007616646784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076935.145262:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616646784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076935.145272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076935.145276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076935.145280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076935.145288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076932/real 1769076935] req@ffff995b8455ad80 x1855007616646912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076935.145302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616646912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076935.145316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076935.145321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076935.145326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616646912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076935.145335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076932/real 1769076935] req@ffff995b8455b480 x1855007616646656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076935.145353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616646656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076935.145365:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076935.145369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076935.145374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616646656:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076935.145488:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076932/real 1769076935] req@ffff995b84558000 x1855007616647424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076935.145511:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616647424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076935.145523:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076935.145529:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076935.145535:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616647424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076935.848551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076935.848630:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076936.168513:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076936.168540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076936.168544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076936.168584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076936.872684:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076936.872728:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076937.192582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076937.192622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076937.192636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769076937.192641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076937.192643:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076937.192659:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076937.192664:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3922 00000100:00080000:3.0:1769076937.192685:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076937.192698:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076937.192703:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076937.192754:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616647552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076937.192787:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076937.192802:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076937.192804:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076937.192810:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3922 00000100:00080000:3.0:1769076937.192814:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076937.192819:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076937.192831:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076937.192849:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616647680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076937.192864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076937.192868:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076937.192871:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076937.192878:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616647552:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076937.192883:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3922 00000100:00080000:3.0:1769076937.192899:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076937.192912:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076937.192921:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076937.192940:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616647808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076937.192952:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076937.192956:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076937.192959:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076937.192975:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3922 00000100:00080000:3.0:1769076937.192980:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076937.192985:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076937.192992:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076937.193007:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616647936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076937.193019:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076937.193023:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076937.193026:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076937.193040:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3922 00000100:00080000:3.0:1769076937.193046:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076937.193055:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769076937.193062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769076937.193068:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076937.193088:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616648064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076937.193102:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076937.193106:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076937.193109:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076937.193115:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3922 00000100:00080000:3.0:1769076937.193121:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076937.193127:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076937.193135:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076937.193152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076937/real 1769076937] req@ffff995b84559500 x1855007616647552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076937.193156:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616648192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076937.193210:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076937.193217:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076937.193228:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076937.193229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616647552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076937.193253:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3922 00000100:00080000:3.0:1769076937.193273:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076937.193279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076937.193287:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769076937.193299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769076937.193304:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076937.193312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616647552:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076937.193333:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616648320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076937.193391:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076937.193438:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076937.193500:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076937/real 1769076937] req@ffff995b84558a80 x1855007616647680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076937.193523:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616647680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076937.193537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076937.193543:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076937.193548:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076937.193559:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076937.193589:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076937.193606:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076937/real 1769076937] req@ffff995b8455a680 x1855007616647808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076937.193635:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616647808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076937.193652:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076937.193657:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076937.193663:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076937.193672:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647936:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076937.193709:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076937.193728:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076937/real 1769076937] req@ffff995b8455b100 x1855007616647936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076937.193746:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616647936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076937.193760:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076937.193766:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076937.193771:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616647936:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076937.193780:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648064:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076937.193808:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076937.193826:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076937/real 1769076937] req@ffff995b8455aa00 x1855007616648064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076937.193843:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616648064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076937.193866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076937.193892:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076937.193898:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648064:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076937.193914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616648192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076937.193946:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076937.193963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076937/real 1769076937] req@ffff995b8455bb80 x1855007616648192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076937.193980:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616648192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076937.193992:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076937.193997:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076937.194002:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616648192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076937.194012:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616648320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076937.194038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076937.194055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076937/real 1769076937] req@ffff995b84558000 x1855007616648320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076937.194073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616648320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076937.194085:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076937.194090:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076937.194097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616648320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076937.896678:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076937.896786:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076938.216560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076938.216562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076938.216616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076938.216701:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076938.920598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076938.920642:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076939.240568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076939.240569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076939.240616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076939.240630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076939.944536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076939.944620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076940.264561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076940.264562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076940.264572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076940.264575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076940.968590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076940.968624:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076941.288556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076941.288611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076941.288654:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076941.288657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076941.992535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076941.992640:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076942.312585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076942.312591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769076942.312634:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076942.312646:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769076942.312651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076942.312653:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076942.312656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076942.312675:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3928 00000100:00080000:3.0:1769076942.312692:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076942.312705:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076942.312761:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616648448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076942.312798:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076942.312804:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076942.312807:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076942.312812:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3928 00000100:00080000:3.0:1769076942.312817:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076942.312823:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076942.312835:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076942.312845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616648448:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076942.312854:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616648576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076942.312870:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076942.312874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076942.312877:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076942.312881:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3928 00000100:00080000:3.0:1769076942.312884:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076942.312889:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076942.312895:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076942.312908:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616648704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076942.312921:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076942.312928:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076942.312933:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076942.312937:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076942.312944:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3928 00000100:00080000:3.0:1769076942.312951:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076942.312956:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076942.312963:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076942.312986:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616648832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076942.312998:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076942.313002:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076942.313005:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076942.313009:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3928 00000100:00080000:3.0:1769076942.313013:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076942.313019:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076942.313026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076942/real 1769076942] req@ffff995b8455b480 x1855007616648448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076942.313030:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076942.313062:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616648960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076942.313077:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616648448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076942.313084:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076942.313123:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076942.313129:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076942.313131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076942.313154:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076942.313157:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3928 00000100:00100000:0.0:1769076942.313165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616648448:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076942.313168:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076942.313185:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076942.313195:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076942.313211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076942.313226:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616649088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076942.313279:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076942.313285:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076942.313289:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076942.313296:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3928 00000100:00080000:3.0:1769076942.313313:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076942.313318:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076942.313325:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076942.313335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076942.313355:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616649216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076942.313366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076942/real 1769076942] req@ffff995b8455ad80 x1855007616648576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076942.313413:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616648576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076942.313438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076942.313444:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076942.313450:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076942.313461:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076942.313520:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076942.313533:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076942/real 1769076942] req@ffff995b84559880 x1855007616648704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076942.313545:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616648704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076942.313554:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076942.313557:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076942.313560:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076942.313577:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648832:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076942.313618:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076942.313631:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076942/real 1769076942] req@ffff995b84559c00 x1855007616648832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076942.313646:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616648832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076942.313656:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076942.313661:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076942.313666:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076942.313674:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648960:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769076942.313694:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076942.313703:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076942/real 1769076942] req@ffff995b8455a300 x1855007616648960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076942.313712:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616648960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076942.313719:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076942.313721:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076942.313724:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616648960:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076942.313741:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616649088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076942.313753:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076942.313760:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076942/real 1769076942] req@ffff995b84558700 x1855007616649088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076942.313769:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616649088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076942.313774:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076942.313776:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076942.313778:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616649088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076942.313783:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616649216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769076942.313793:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076942.313801:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076942/real 1769076942] req@ffff995b8455b480 x1855007616649216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076942.313809:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616649216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076942.313815:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076942.313817:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769076942.313819:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616649216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076943.016631:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076943.016768:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076943.336650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076943.336702:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076943.336750:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076943.336775:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076944.040555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076944.040743:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076944.360901:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076944.360951:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076944.360987:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076944.361011:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076945.064595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076945.064726:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076945.384532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076945.384553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076945.384611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076945.384672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076946.088525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076946.088580:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076946.408548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076946.408567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076946.408586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076946.408604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076947.112551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076947.112613:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076947.432528:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076947.432535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076947.432556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076947.432575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076947.432581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076947.432585:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076947.432592:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3933 00000100:00080000:1.0:1769076947.432601:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076947.432607:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076947.432656:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616649344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076947.432694:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076947.432700:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076947.432703:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076947.432711:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3933 00000100:00080000:1.0:1769076947.432718:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076947.432725:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076947.432738:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076947.432743:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616649344:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769076947.432761:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616649472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076947.432789:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076947.432793:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076947.432796:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076947.432800:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3933 00000100:00080000:1.0:1769076947.432807:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076947.432811:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076947.432817:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076947.432830:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076947.432837:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616649600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076947.432869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076947.432875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076947.432878:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076947.432883:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3933 00000100:00080000:1.0:1769076947.432889:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076947.432894:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076947/real 1769076947] req@ffff995b868c7800 x1855007616649344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076947.432900:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076947.432913:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076947.432922:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616649344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076947.432938:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076947.432943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769076947.432951:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616649728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076947.432964:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076947.432964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076947.432965:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616649344:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076947.432997:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076947.433001:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076947.433009:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3933 00000100:00080000:1.0:1769076947.433017:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076947.433023:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076947.433030:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076947.433036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616649472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769076947.433044:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616649856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076947.433073:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076947.433077:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076947.433079:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076947.433085:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3933 00000800:00000100:2.0:1769076947.433092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076947.433095:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076947.433103:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076947.433109:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076947.433114:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076947/real 1769076947] req@ffff995b868c4a80 x1855007616649472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076947.433124:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616649984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076947.433137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616649472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076947.433163:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076947.433164:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076947.433186:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076947.433187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769076947.433191:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076947.433197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616649472:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076947.433199:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3933 00000100:00080000:1.0:1769076947.433207:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076947.433210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616649600:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076947.433214:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076947.433232:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076947.433265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769076947.433267:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616650112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076947.433282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076947/real 1769076947] req@ffff995b868c6300 x1855007616649600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076947.433316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616649600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076947.433327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076947.433332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076947.433336:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616649600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076947.433343:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616649728:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076947.433370:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076947.433379:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076947/real 1769076947] req@ffff995b868c6d80 x1855007616649728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076947.433389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616649728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076947.433397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076947.433399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076947.433402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616649728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076947.433417:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616649856:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076947.433431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076947.433455:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076947/real 1769076947] req@ffff995b868c5500 x1855007616649856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076947.433463:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616649856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076947.433475:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076947.433481:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076947.433506:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616649856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076947.433511:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616649984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076947.433525:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076947.433533:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076947/real 1769076947] req@ffff995b868c4700 x1855007616649984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076947.433541:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616649984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076947.433547:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076947.433549:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076947.433552:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616649984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076947.433555:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616650112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076947.433565:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076947.433571:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076947/real 1769076947] req@ffff995b868c7480 x1855007616650112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076947.433579:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616650112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076947.433585:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076947.433587:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076947.433589:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616650112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076948.136588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076948.136699:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076948.456527:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076948.456527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076948.456575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076948.456578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076949.160621:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076949.160703:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076949.480554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076949.480575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076949.480589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076949.480619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076950.184545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076950.184618:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076950.504681:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076950.504692:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076950.504698:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076950.504703:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076951.208538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076951.208621:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076951.528548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076951.528556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076951.528583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076951.528609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076952.232568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076952.232682:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076952.552622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076952.552635:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076952.552640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076952.552665:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076952.552691:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076952.552700:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076952.552705:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076952.552718:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3938 00000100:00080000:0.0:1769076952.552740:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076952.552753:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076952.552811:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616650240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076952.552859:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076952.552867:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076952.552871:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076952.552880:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3938 00000100:00080000:0.0:1769076952.552890:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076952.552901:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076952.552919:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076952.552929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616650240:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076952.552954:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616650368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076952.552979:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076952.552984:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076952.552987:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076952.552993:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3938 00000100:00080000:0.0:1769076952.552999:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076952.553005:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076952.553013:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076952.553043:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616650496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076952.553052:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076952.553114:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076952.553122:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076952.553125:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076952.553144:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3938 00000100:00080000:0.0:1769076952.553153:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076952.553163:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076952.553177:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076952.553179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076952/real 1769076952] req@ffff995b84640700 x1855007616650240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076952.553211:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616650624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076952.553224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616650240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076952.553245:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076952.553259:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076952.553263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076952.553264:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076952.553283:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3938 00000100:00080000:2.0:1769076952.553286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076952.553291:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076952.553300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616650240:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076952.553306:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076952.553323:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076952.553348:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616650752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076952.553356:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616650368:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076952.553392:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076952.553400:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076952.553413:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076952.553423:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076952.553430:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3938 00000100:00080000:0.0:1769076952.553448:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076952.553496:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076952/real 1769076952] req@ffff995b84643800 x1855007616650368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076952.553505:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076952.553520:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076952.553552:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616650880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076952.553555:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616650368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076952.553610:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076952.553612:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076952.553628:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076952.553629:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076952.553641:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076952.553644:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616650368:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076952.553659:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3938 00000100:00100000:2.0:1769076952.553668:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616650496:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076952.553673:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076952.553697:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076952.553706:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076952.553724:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769076952.553737:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616651008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076952.553755:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076952/real 1769076952] req@ffff995b84642a00 x1855007616650496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076952.553786:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616650496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076952.553811:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076952.553819:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076952.553826:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616650496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076952.553838:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616650624:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076952.553870:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076952.553889:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076952/real 1769076952] req@ffff995b84641180 x1855007616650624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076952.553912:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616650624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076952.553928:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076952.553935:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076952.553942:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616650624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076952.553988:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616650752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076952.554030:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076952.554055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076952/real 1769076952] req@ffff995b84641500 x1855007616650752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076952.554081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616650752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076952.554123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076952.554128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076952.554133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616650752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076952.554148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616650880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076952.554179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076952.554195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076952/real 1769076952] req@ffff995b84643b80 x1855007616650880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076952.554219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616650880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076952.554230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076952.554234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076952.554238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616650880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076952.554245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616651008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076952.554262:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076952.554273:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076952/real 1769076952] req@ffff995b84641880 x1855007616651008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076952.554296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616651008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076952.554306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076952.554309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076952.554313:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616651008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076953.256613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076953.256711:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076953.576612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076953.576658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076953.576747:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076953.576749:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076954.280550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076954.280618:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076954.600534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076954.600535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076954.600606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076954.600619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076955.304544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076955.304667:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076955.624551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076955.624556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076955.624663:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076955.624669:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076956.328544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076956.328651:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076956.648612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076956.648618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076956.648618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076956.648620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076957.352542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076957.352617:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076957.672655:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076957.672655:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076957.672669:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076957.672693:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076957.672709:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076957.672709:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076957.672718:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076957.672722:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3943 00000100:00080000:0.0:1769076957.672749:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076957.672768:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076957.672798:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616651136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076957.672823:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076957.672825:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076957.672827:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076957.672830:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3943 00000100:00080000:0.0:1769076957.672834:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076957.672838:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076957.672845:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076957.672856:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616651264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076957.672868:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076957.672870:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616651136:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076957.672872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076957.672875:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076957.672887:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3943 00000100:00080000:0.0:1769076957.672891:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076957.672894:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076957.672899:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076957.672911:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616651392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076957.672919:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076957.672922:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076957.672924:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076957.672930:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3943 00000800:00000100:3.0:1769076957.672934:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076957.672937:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076957.672942:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076957.672947:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076957.672956:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616651520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076957.672963:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076957.672965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076957.672966:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076957.672968:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3943 00000100:00080000:0.0:1769076957.672970:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076957.672972:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076957.672974:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076957.672987:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616651648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076957.672996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076957.673012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076957.673014:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076957.673019:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3943 00000100:00080000:0.0:1769076957.673033:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076957.673038:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076957.673045:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076957.673056:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076957/real 1769076957] req@ffff995b84642d80 x1855007616651136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076957.673060:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616651776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076957.673078:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076957.673081:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076957.673083:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076957.673086:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3943 00000100:00080000:0.0:1769076957.673088:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076957.673091:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076957.673095:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076957.673103:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616651904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076957.673179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616651136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076957.673189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076957.673194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076957.673198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616651136:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769076957.673261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616651264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076957.673286:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076957.673301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076957/real 1769076957] req@ffff995b84643100 x1855007616651264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076957.673313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616651264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076957.673322:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076957.673325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076957.673329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616651264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076957.673335:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616651392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076957.673348:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076957.673357:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076957/real 1769076957] req@ffff995b84643480 x1855007616651392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076957.673365:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616651392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076957.673376:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076957.673382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076957.673386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616651392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076957.673394:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616651520:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076957.673417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076957.673429:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076957/real 1769076957] req@ffff995b84641880 x1855007616651520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076957.673441:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616651520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076957.673450:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076957.673477:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076957.673482:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616651520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076957.673490:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616651648:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076957.673512:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076957.673523:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076957/real 1769076957] req@ffff995b84643b80 x1855007616651648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076957.673535:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616651648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076957.673544:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076957.673548:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076957.673550:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616651648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076957.673555:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616651776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076957.673569:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076957.673576:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076957/real 1769076957] req@ffff995b84641500 x1855007616651776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076957.673583:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616651776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076957.673588:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076957.673590:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076957.673592:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616651776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076957.673596:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616651904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076957.673605:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076957.673610:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076957/real 1769076957] req@ffff995b84641180 x1855007616651904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076957.673618:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616651904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076957.673623:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076957.673624:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076957.673626:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616651904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076958.376551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076958.376616:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076958.696559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076958.696561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076958.696603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076958.696604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076959.400543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076959.400592:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076959.720569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076959.720569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076959.720570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076959.720579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076960.424568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076960.424691:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076960.744588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076960.744593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076960.744600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076960.744635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076961.448554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076961.448623:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076961.768566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076961.768566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076961.768573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076961.768590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076962.472556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076962.472638:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076962.792543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076962.792547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769076962.792575:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076962.792584:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769076962.792585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769076962.792589:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076962.792603:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3948 00000100:00080000:1.0:1769076962.792615:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076962.792621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076962.792625:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076962.792661:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616652032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076962.792690:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076962.792694:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076962.792697:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076962.792703:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3948 00000100:00080000:1.0:1769076962.792709:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076962.792714:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076962.792720:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616652032:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769076962.792725:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076962.792745:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616652160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076962.792755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076962.792757:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076962.792759:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076962.792762:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3948 00000100:00080000:1.0:1769076962.792766:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076962.792769:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076962.792773:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076962.792782:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616652288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076962.792793:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076962.792795:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076962.792797:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076962.792800:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3948 00000800:00000100:3.0:1769076962.792803:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769076962.792804:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076962.792809:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076962.792813:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769076962.792821:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616652416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076962.792829:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076962.792832:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076962.792833:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076962.792836:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3948 00000100:00080000:1.0:1769076962.792839:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076962.792842:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076962.792846:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769076962.792854:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076962/real 1769076962] req@ffff995b868c5c00 x1855007616652032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076962.792857:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616652544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076962.792876:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076962.792878:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076962.792879:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076962.792887:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3948 00000100:00080000:1.0:1769076962.792891:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076962.792895:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076962.792899:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616652032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076962.792903:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076962.792937:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076962.792943:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616652672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076962.792944:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076962.792952:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616652032:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769076962.792968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076962.792972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076962.792975:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076962.792980:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3948 00000100:00080000:1.0:1769076962.792984:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076962.792986:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616652160:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769076962.792992:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769076962.792998:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076962.793011:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616652800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076962.793019:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076962.793038:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076962/real 1769076962] req@ffff995b868c6a00 x1855007616652160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076962.793048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616652160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076962.793055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076962.793058:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076962.793060:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616652160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076962.793065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616652288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076962.793084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076962.793092:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076962/real 1769076962] req@ffff995b868c7100 x1855007616652288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076962.793100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616652288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076962.793107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076962.793109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076962.793111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616652288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076962.793116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616652416:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076962.793127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076962.793133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076962/real 1769076962] req@ffff995b868c7480 x1855007616652416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076962.793141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616652416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076962.793147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076962.793149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076962.793151:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616652416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076962.793157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616652544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076962.793176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076962.793188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076962/real 1769076962] req@ffff995b868c4700 x1855007616652544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076962.793212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616652544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076962.793223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076962.793226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076962.793230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616652544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076962.793237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616652672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076962.793258:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076962.793272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076962/real 1769076962] req@ffff995b868c5500 x1855007616652672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076962.793288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616652672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076962.793296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076962.793298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076962.793301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616652672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076962.793315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616652800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076962.793331:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076962.793339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076962/real 1769076962] req@ffff995b868c6d80 x1855007616652800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076962.793348:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616652800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076962.793358:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076962.793360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076962.793363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616652800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076963.496601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076963.496699:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076963.816554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076963.816570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076963.816585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076963.816612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076964.520592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076964.520623:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076964.840556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076964.840585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076964.840612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076964.840621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076965.544513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076965.544593:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076965.864604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076965.864612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076965.864634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076965.864689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076966.568525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076966.568621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076966.888565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076966.888569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076966.888580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076966.888593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076967.592542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076967.592734:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076967.912565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076967.912599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076967.912603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076967.912612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076967.912618:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076967.912622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076967.912641:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3953 00000100:00080000:0.0:1769076967.912656:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076967.912666:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769076967.912683:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076967.912724:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616652928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076967.912750:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076967.912754:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076967.912756:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076967.912760:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3953 00000100:00080000:0.0:1769076967.912764:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076967.912769:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076967.912778:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076967.912791:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616652928:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076967.912816:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616653056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076967.912839:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076967.912843:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076967.912844:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076967.912848:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3953 00000100:00080000:0.0:1769076967.912852:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076967.912856:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076967.912860:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076967.912889:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616653184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076967.912898:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076967.912902:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076967.912906:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769076967.912908:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076967.912913:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3953 00000100:00080000:0.0:1769076967.912920:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076967.912925:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076967.912929:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076967.912960:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616653312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076967.912970:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076967.912974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076967.912976:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076967.912980:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3953 00000100:00080000:0.0:1769076967.912985:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076967.912993:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076967/real 1769076967] req@ffff995b84642a00 x1855007616652928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076967.912994:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076967.913003:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076967.913032:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616653440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076967.913048:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076967.913052:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076967.913054:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076967.913057:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3953 00000100:00080000:0.0:1769076967.913062:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076967.913066:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076967.913071:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076967.913077:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616652928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076967.913090:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616653568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076967.913116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076967.913138:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076967.913142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076967.913146:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076967.913149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616652928:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076967.913151:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076967.913158:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3953 00000100:00080000:0.0:1769076967.913164:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076967.913169:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076967.913174:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076967.913179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076967.913186:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616653696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076967.913215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076967.913235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076967/real 1769076967] req@ffff995b84643800 x1855007616653056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076967.913249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616653056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076967.913257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076967.913259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076967.913262:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076967.913267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076967.913302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076967.913312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076967/real 1769076967] req@ffff995b84640700 x1855007616653184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076967.913322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616653184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076967.913331:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076967.913334:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076967.913337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076967.913342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653312:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076967.913355:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076967.913362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076967/real 1769076967] req@ffff995b84642680 x1855007616653312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076967.913372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616653312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076967.913380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076967.913382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076967.913385:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076967.913389:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769076967.913401:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076967.913408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076967/real 1769076967] req@ffff995b84641c00 x1855007616653440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076967.913432:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616653440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076967.913438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076967.913440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076967.913442:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769076967.913445:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616653568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076967.913456:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076967.913462:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076967/real 1769076967] req@ffff995b84640000 x1855007616653568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076967.913469:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616653568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076967.913474:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076967.913476:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076967.913478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616653568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076967.913526:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616653696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769076967.913541:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076967.913549:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076967/real 1769076967] req@ffff995b84640e00 x1855007616653696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076967.913557:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616653696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076967.913563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076967.913565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769076967.913568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616653696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076968.616546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076968.616608:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076968.936571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076968.936581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076968.936584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076968.936599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076969.640573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076969.640667:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076969.960636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076969.960639:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076969.960651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076969.960773:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076970.664622:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076970.664723:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076970.984521:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076970.984536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076970.984545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076970.984581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076971.688597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076971.688826:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076972.008553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076972.008553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076972.008565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769076972.008588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076972.712594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076972.712667:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076973.032537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076973.032570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769076973.032614:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076973.032634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769076973.032637:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076973.032646:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076973.032648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076973.032661:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3958 00000100:00080000:3.0:1769076973.032678:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076973.032687:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769076973.032737:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616653824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076973.032791:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076973.032794:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076973.032797:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076973.032800:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3958 00000100:00080000:3.0:1769076973.032804:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076973.032817:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076973.032829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616653824:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769076973.032830:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076973.032853:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616653952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076973.032867:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076973.032872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076973.032874:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076973.032877:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3958 00000100:00080000:3.0:1769076973.032892:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076973.032898:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076973.032904:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076973.032910:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769076973.032921:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616654080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076973.032935:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076973.032939:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076973.032941:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076973.032947:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3958 00000100:00080000:3.0:1769076973.032952:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076973.032960:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076973/real 1769076973] req@ffff995b84558000 x1855007616653824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076973.032961:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076973.032970:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076973.032983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616653824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076973.032986:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616654208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076973.032995:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076973.033004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076973.033012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616653824:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769076973.033013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076973.033036:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076973.033038:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076973.033042:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3958 00000100:00080000:3.0:1769076973.033045:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076973.033049:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076973.033053:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076973.033058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076973.033075:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616654336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076973.033085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769076973.033086:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076973.033089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076973.033092:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076973.033097:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3958 00000100:00080000:3.0:1769076973.033101:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076973.033103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076973/real 1769076973] req@ffff995b8455bb80 x1855007616653952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076973.033107:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769076973.033114:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076973.033120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616653952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076973.033127:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616654464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076973.033132:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076973.033141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769076973.033142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076973.033145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616653952:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076973.033151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076973.033156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654080:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769076973.033156:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076973.033167:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3958 00000100:00080000:3.0:1769076973.033174:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076973.033177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769076973.033183:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076973.033189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076973/real 1769076973] req@ffff995b8455aa00 x1855007616654080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076973.033193:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076973.033205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616654080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076973.033215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076973.033219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076973.033222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769076973.033222:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616654592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076973.033230:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654208:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076973.033253:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076973.033262:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076973/real 1769076973] req@ffff995b8455b100 x1855007616654208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076973.033271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616654208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076973.033279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076973.033281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076973.033284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654208:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076973.033294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654336:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076973.033304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076973.033312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076973/real 1769076973] req@ffff995b8455a680 x1855007616654336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076973.033333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616654336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076973.033353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076973.033357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076973.033361:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076973.033367:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616654464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076973.033392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076973.033405:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076973/real 1769076973] req@ffff995b84558a80 x1855007616654464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076973.033416:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616654464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076973.033435:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076973.033439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076973.033443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616654464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076973.033450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616654592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076973.033509:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076973.033520:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076973/real 1769076973] req@ffff995b84559500 x1855007616654592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076973.033528:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616654592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076973.033535:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076973.033538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076973.033541:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616654592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076973.736584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076973.736669:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076974.056612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076974.056612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076974.056646:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076974.056659:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076974.760568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076974.760695:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076975.080606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076975.080613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076975.080637:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076975.080689:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076975.784576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076975.784656:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076976.104533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076976.104576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076976.104609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076976.104652:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076976.808587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076976.808652:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076977.128663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076977.128663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076977.128663:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076977.128736:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076977.832573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076977.832697:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076978.152590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076978.152597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076978.152615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076978.152616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076978.152676:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076978.152683:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076978.152688:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076978.152715:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3963 00000100:00080000:0.0:1769076978.152726:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076978.152733:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076978.152788:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616654720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076978.152809:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076978.152812:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076978.152814:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076978.152817:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3963 00000100:00080000:0.0:1769076978.152820:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076978.152824:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076978.152832:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076978.152839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616654720:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076978.152860:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616654848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076978.152877:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076978.152880:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076978.152881:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076978.152884:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3963 00000100:00080000:0.0:1769076978.152888:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076978.152890:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076978.152894:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076978.152905:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616654976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076978.152917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076978.152921:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076978.152942:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076978.152942:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076978.152951:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3963 00000100:00080000:0.0:1769076978.152956:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076978.152961:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076978.152967:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076978.152978:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616655104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076978.152985:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076978.152990:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769076978.153006:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076978/real 1769076978] req@ffff995b84640380 x1855007616654720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076978.153008:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076978.153014:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3963 00000100:00080000:0.0:1769076978.153020:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076978.153025:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076978.153030:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076978.153042:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616654720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076978.153043:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616655232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076978.153066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076978.153069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076978.153070:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076978.153072:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076978.153074:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3963 00000100:00080000:0.0:1769076978.153078:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076978.153080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076978.153084:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076978.153088:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616654720:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769076978.153090:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076978.153100:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616655360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076978.153122:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076978.153124:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076978.153125:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076978.153130:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3963 00000100:00100000:2.0:1769076978.153130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654848:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076978.153135:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076978.153139:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076978.153143:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076978.153153:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616655488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076978.153163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076978.153180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076978/real 1769076978] req@ffff995b84641f80 x1855007616654848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076978.153191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616654848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076978.153197:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076978.153200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076978.153203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076978.153208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076978.153223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076978.153234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076978/real 1769076978] req@ffff995b84640a80 x1855007616654976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076978.153248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616654976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076978.153259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076978.153263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076978.153277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616654976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076978.153285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616655104:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076978.153319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076978.153327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076978/real 1769076978] req@ffff995b84640e00 x1855007616655104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076978.153342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616655104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076978.153348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076978.153351:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076978.153353:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616655104:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076978.153357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616655232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076978.153374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076978.153385:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076978/real 1769076978] req@ffff995b84640000 x1855007616655232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076978.153398:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616655232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076978.153408:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076978.153411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076978.153415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616655232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076978.153421:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616655360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076978.153437:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076978.153445:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076978/real 1769076978] req@ffff995b84641c00 x1855007616655360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076978.153471:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616655360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076978.153477:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076978.153479:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076978.153481:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616655360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076978.153502:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616655488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076978.153540:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076978.153554:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076978/real 1769076978] req@ffff995b84642680 x1855007616655488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076978.153568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616655488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076978.153576:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076978.153579:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076978.153581:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616655488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076978.856557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076978.856671:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076979.176610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076979.176610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076979.176618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076979.176705:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076979.880567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076979.880648:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076980.200582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076980.200608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076980.200617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076980.200661:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076980.904549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076980.904652:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076981.224555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076981.224557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076981.224665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076981.224674:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076981.928615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076981.928721:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076982.248561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076982.248564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076982.248573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076982.248606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076982.952562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076982.952711:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076983.272664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076983.272664:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076983.272671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076983.272673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076983.272739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076983.272747:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076983.272753:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076983.272765:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3968 00000100:00080000:0.0:1769076983.272788:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076983.272796:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076983.272834:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616655616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076983.272862:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076983.272866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076983.272868:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076983.272872:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3968 00000100:00080000:0.0:1769076983.272875:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076983.272879:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076983.272889:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076983.272902:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616655744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076983.272906:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616655616:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076983.272940:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076983.272945:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076983.272947:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076983.272952:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3968 00000100:00080000:0.0:1769076983.272957:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076983.272963:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076983.272970:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076983.272988:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616655872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076983.273023:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076983.273024:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076983.273030:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076983.273033:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076983.273039:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3968 00000100:00080000:0.0:1769076983.273045:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076983.273050:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076983.273057:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076983.273074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076983/real 1769076983] req@ffff995b84640700 x1855007616655616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076983.273081:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616656000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076983.273098:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076983.273103:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769076983.273104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616655616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076983.273106:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076983.273115:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3968 00000100:00080000:0.0:1769076983.273122:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076983.273125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076983.273127:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769076983.273133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769076983.273134:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076983.273138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616655616:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076983.273147:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616656128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076983.273154:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076983.273157:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076983.273159:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076983.273164:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3968 00000100:00100000:2.0:1769076983.273164:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616655744:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076983.273170:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076983.273174:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076983.273178:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076983.273189:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616656256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076983.273195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076983.273207:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076983.273210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076983/real 1769076983] req@ffff995b84643800 x1855007616655744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076983.273212:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076983.273216:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076983.273223:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3968 00000100:00080000:0.0:1769076983.273229:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076983.273235:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076983.273243:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076983.273247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616655744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076983.273256:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616656384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076983.273257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076983.273266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076983.273274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616655744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076983.273282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616655872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076983.273298:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076983.273306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076983/real 1769076983] req@ffff995b84642a00 x1855007616655872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076983.273314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616655872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076983.273321:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076983.273323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076983.273326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616655872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076983.273330:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656000:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076983.273341:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076983.273347:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076983/real 1769076983] req@ffff995b84641180 x1855007616656000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076983.273355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616656000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076983.273366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076983.273369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076983.273371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076983.273375:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656128:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076983.273384:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076983.273390:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076983/real 1769076983] req@ffff995b84641500 x1855007616656128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076983.273398:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616656128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076983.273416:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076983.273418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076983.273420:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076983.273434:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616656256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076983.273447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076983.273476:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076983/real 1769076983] req@ffff995b84643b80 x1855007616656256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076983.273485:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616656256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076983.273491:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076983.273494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076983.273497:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616656256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076983.273501:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616656384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076983.273515:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076983.273522:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076983/real 1769076983] req@ffff995b84641880 x1855007616656384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076983.273530:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616656384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076983.273536:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076983.273538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076983.273540:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616656384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076983.976530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076983.976648:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076984.296645:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076984.296652:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076984.296706:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076984.296735:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076985.000564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076985.000648:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076985.320571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076985.320574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076985.320584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076985.320592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076986.024563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076986.024655:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076986.344587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076986.344594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076986.344598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076986.344626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076987.048604:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076987.048711:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076987.368584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076987.368593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769076987.368620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076987.368643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076988.072564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076988.072680:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076988.392596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076988.392665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076988.392674:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076988.392756:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076988.392762:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076988.392774:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076988.392781:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076988.392794:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3974 00000100:00080000:0.0:1769076988.392808:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076988.392818:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076988.392869:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616656512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076988.392907:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076988.392912:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076988.392915:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076988.392920:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3974 00000100:00080000:0.0:1769076988.392925:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076988.392930:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076988.392942:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076988.392960:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616656640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076988.392969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616656512:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076988.392994:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076988.392999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076988.393003:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076988.393009:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3974 00000100:00080000:0.0:1769076988.393013:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076988.393018:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076988.393024:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076988.393042:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616656768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076988.393055:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076988.393060:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076988.393062:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076988.393066:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3974 00000100:00080000:0.0:1769076988.393071:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076988.393074:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076988.393083:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076988.393092:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076988.393110:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616656896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076988.393122:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076988.393127:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076988.393130:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076988.393141:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3974 00000100:00100000:2.0:1769076988.393143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076988/real 1769076988] req@ffff995b84643480 x1855007616656512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076988.393146:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076988.393157:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076988.393168:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769076988.393187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616656512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076988.393194:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616657024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076988.393232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076988.393235:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076988.393246:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076988.393247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076988.393253:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076988.393277:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3974 00000100:00100000:2.0:1769076988.393277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616656512:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076988.393287:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076988.393297:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076988.393303:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076988.393315:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616657152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076988.393326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656640:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076988.393342:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076988.393349:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076988.393352:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076988.393360:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3974 00000100:00080000:0.0:1769076988.393368:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076988.393376:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076988.393380:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076988.393391:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076988.393403:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076988/real 1769076988] req@ffff995b84643100 x1855007616656640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076988.393413:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616657280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076988.393446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616656640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076988.393535:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076988.393543:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076988.393551:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076988.393564:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076988.393609:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076988.393632:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076988/real 1769076988] req@ffff995b84642d80 x1855007616656768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076988.393650:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616656768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076988.393670:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076988.393677:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076988.393683:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076988.393696:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076988.393739:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076988.393763:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076988/real 1769076988] req@ffff995b84641880 x1855007616656896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076988.393789:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616656896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076988.393801:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076988.393805:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076988.393809:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616656896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076988.393818:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657024:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076988.393845:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076988.393859:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076988/real 1769076988] req@ffff995b84643b80 x1855007616657024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076988.393873:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616657024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076988.393885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076988.393888:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076988.393892:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076988.393917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616657152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076988.393936:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076988.393947:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076988/real 1769076988] req@ffff995b84641500 x1855007616657152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076988.393961:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616657152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076988.393972:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076988.393976:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076988.393980:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616657152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076988.393986:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616657280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076988.394004:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076988.394024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076988/real 1769076988] req@ffff995b84641180 x1855007616657280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076988.394037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616657280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076988.394047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076988.394050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076988.394054:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616657280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076989.096561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076989.096722:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076989.416584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076989.416595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076989.416684:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076989.416722:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076990.120601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076990.120697:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076990.440643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076990.440709:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076990.440749:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076990.440792:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076991.144610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076991.144701:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076991.464613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076991.464626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076991.464697:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769076991.464698:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076992.168590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076992.168692:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076992.488595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076992.488600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076992.488621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769076992.488622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076993.192530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076993.192598:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076993.512574:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076993.512582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076993.512583:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769076993.512586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076993.512588:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076993.512598:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3979 00000100:00080000:0.0:1769076993.512628:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076993.512641:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076993.512641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769076993.512692:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076993.512718:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616657408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076993.512761:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076993.512764:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076993.512766:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076993.512770:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3979 00000100:00080000:0.0:1769076993.512774:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076993.512778:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076993.512786:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076993.512792:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616657408:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076993.512817:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616657536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076993.512830:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076993.512834:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076993.512836:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076993.512840:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3979 00000100:00080000:0.0:1769076993.512845:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076993.512849:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076993.512854:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076993.512885:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616657664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076993.512896:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076993.512904:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1769076993.512906:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769076993.512907:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076993.512919:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3979 00000100:00080000:0.0:1769076993.512925:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076993.512930:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076993.512954:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076993.512963:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616657792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076993.512971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076993.512974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076993.512975:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076993.512979:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3979 00000100:00080000:0.0:1769076993.512982:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076993.512987:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769076993.512988:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076993/real 1769076993] req@ffff995b84642a00 x1855007616657408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076993.512992:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076993.513014:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616657920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076993.513021:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616657408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076993.513027:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076993.513036:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076993.513039:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076993.513044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076993.513047:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3979 00000100:00080000:2.0:1769076993.513053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769076993.513054:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076993.513062:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616657408:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769076993.513063:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076993.513074:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076993.513088:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616658048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076993.513096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657536:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076993.513098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076993.513124:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076993.513126:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076993.513130:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3979 00000100:00080000:0.0:1769076993.513134:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076993.513138:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076993.513144:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076993.513158:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616658176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076993.513167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076993.513217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076993/real 1769076993] req@ffff995b84643800 x1855007616657536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076993.513252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616657536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076993.513259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076993.513261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076993.513264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076993.513268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076993.513283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076993.513290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076993/real 1769076993] req@ffff995b84640700 x1855007616657664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076993.513299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616657664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076993.513331:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076993.513335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076993.513339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076993.513345:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076993.513364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076993.513373:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076993/real 1769076993] req@ffff995b84642680 x1855007616657792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076993.513386:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616657792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076993.513419:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076993.513424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076993.513427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076993.513449:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769076993.513488:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076993.513506:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076993/real 1769076993] req@ffff995b84641c00 x1855007616657920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076993.513522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616657920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076993.513533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076993.513539:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076993.513544:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616657920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076993.513572:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616658048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076993.513618:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076993.513634:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076993/real 1769076993] req@ffff995b84640000 x1855007616658048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076993.513650:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616658048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076993.513660:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076993.513664:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076993.513685:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616658048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076993.513691:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616658176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769076993.513706:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769076993.513715:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076993/real 1769076993] req@ffff995b84640e00 x1855007616658176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076993.513725:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616658176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076993.513732:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076993.513752:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769076993.513754:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616658176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076994.216550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076994.216802:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076994.536609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769076994.536635:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076994.536642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076994.536669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769076995.240589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076995.240705:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076995.560559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076995.560562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076995.560564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076995.560590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076996.264562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076996.264630:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076996.584559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076996.584576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769076996.584579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769076996.584620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076997.288619:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076997.288726:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076997.608630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076997.608640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769076997.608685:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769076997.608713:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769076998.312595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076998.312731:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076998.632544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769076998.632550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769076998.632576:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076998.632583:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076998.632589:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076998.632600:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3984 00000100:00100000:3.0:1769076998.632607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769076998.632614:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076998.632618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769076998.632625:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769076998.632674:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616658304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076998.632709:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076998.632713:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076998.632715:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076998.632719:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3984 00000100:00080000:0.0:1769076998.632722:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076998.632726:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076998.632735:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076998.632738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616658304:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769076998.632750:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616658432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076998.632759:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076998.632762:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076998.632774:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076998.632777:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3984 00000100:00080000:0.0:1769076998.632779:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076998.632783:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076998.632787:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076998.632796:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616658560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076998.632803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076998.632806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076998.632807:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076998.632809:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3984 00000100:00080000:0.0:1769076998.632813:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076998.632815:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076998.632819:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076998.632827:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616658688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076998.632833:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076998.632836:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076998.632837:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076998.632840:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3984 00000100:00080000:0.0:1769076998.632843:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076998.632845:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076998.632848:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769076998.632858:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616658816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076998.632865:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076998.632868:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076998.632869:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076998.632877:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616658432:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076998.632884:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3984 00000100:00080000:0.0:1769076998.632888:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076998.632891:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076998.632895:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076998.632905:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616658944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076998.632911:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616658560:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769076998.632917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076998.632920:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076998.632922:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076998.632926:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3984 00000100:00080000:0.0:1769076998.632930:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076998.632933:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769076998.632955:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076998.632969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616658688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769076998.632976:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616659072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076998.633035:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616658816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769076998.633064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616658944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769076998.633093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616659072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769076999.336652:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076999.336803:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076999.656653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769076999.656761:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769076999.656764:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769076999.656765:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077000.360554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077000.360760:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077000.680644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077000.680734:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077000.680745:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077000.680805:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077001.384622:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077001.384713:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077001.704642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077001.704688:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077001.704714:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077001.704863:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:2.0:1769077001.704994:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769077001.705046:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077001.705064:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077001.705103:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077001.705117:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077001.705127:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769077001.705136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076998/real 1769077001] req@ffff995b84641f80 x1855007616658432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1769077001.705137:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077001.705148:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077001.705168:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769077001.705169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616658432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077001.705185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077001.705192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077001.705198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616658432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077001.705209:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076998/real 1769077001] req@ffff995b84640380 x1855007616658560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077001.705228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616658560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077001.705237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077001.705241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077001.705246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616658560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077001.705256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076998/real 1769077001] req@ffff995b84640e00 x1855007616658688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077001.705327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616658688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077001.705341:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077001.705347:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077001.705352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616658688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077001.705362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076998/real 1769077001] req@ffff995b84640000 x1855007616658816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077001.705380:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616658816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077001.705392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077001.705398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077001.705403:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616658816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077001.705413:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076998/real 1769077001] req@ffff995b84640a80 x1855007616658304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077001.705430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616658304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077001.705440:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077001.705444:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077001.705448:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616658304:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077001.705547:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076998/real 1769077001] req@ffff995b84641c00 x1855007616658944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077001.705568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616658944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077001.705582:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077001.705588:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077001.705595:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616658944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077001.705605:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076998/real 1769077001] req@ffff995b84642680 x1855007616659072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077001.705624:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616659072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077001.705637:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077001.705642:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077001.705647:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616659072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077002.408538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077002.408618:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077002.728602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077002.728603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077002.728630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077002.728675:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077003.432567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077003.432730:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077003.752583:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077003.752584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077003.752597:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077003.752603:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077003.752643:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3989 00000100:00100000:1.0:1769077003.752654:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077003.752660:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077003.752671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077003.752676:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077003.752685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077003.752720:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616659200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077003.752762:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077003.752768:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077003.752771:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077003.752779:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3989 00000100:00080000:0.0:1769077003.752785:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077003.752793:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077003.752806:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077003.752842:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616659200:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077003.752847:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616659328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077003.752870:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077003.752877:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077003.752896:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077003.752901:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3989 00000100:00080000:0.0:1769077003.752906:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077003.752912:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077003.752919:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077003.752942:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616659456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077003.752951:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077003.752964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077003.752969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077003.752972:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077003.752979:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3989 00000100:00080000:0.0:1769077003.752984:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077003.752989:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077003.752997:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077003.753007:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077003/real 1769077003] req@ffff995b84640700 x1855007616659200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077003.753016:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616659584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077003.753036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077003.753043:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769077003.753044:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616659200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077003.753046:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077003.753056:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3989 00000100:00080000:0.0:1769077003.753065:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077003.753070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077003.753073:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077003.753081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769077003.753083:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077003.753097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616659200:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077003.753101:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616659712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077003.753116:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077003.753122:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077003.753124:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077003.753129:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3989 00000100:00080000:0.0:1769077003.753134:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077003.753137:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616659328:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077003.753144:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077003.753152:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077003.753169:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616659840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077003.753181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077003.753185:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077003.753190:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077003.753193:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077003.753199:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3989 00000100:00080000:0.0:1769077003.753207:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077003.753213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077003/real 1769077003] req@ffff995b84643800 x1855007616659328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077003.753214:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077003.753241:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077003.753256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616659328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077003.753263:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616659968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077003.753287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077003.753296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077003.753303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616659328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077003.753313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616659456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077003.753340:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077003.753354:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077003/real 1769077003] req@ffff995b84642a00 x1855007616659456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077003.753368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616659456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077003.753382:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077003.753386:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077003.753390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616659456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077003.753397:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616659584:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077003.753419:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077003.753431:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077003/real 1769077003] req@ffff995b84641180 x1855007616659584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077003.753444:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616659584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077003.753481:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077003.753486:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077003.753490:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616659584:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077003.753498:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616659712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077003.753523:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077003.753657:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077003/real 1769077003] req@ffff995b84641500 x1855007616659712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077003.753673:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616659712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077003.753684:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077003.753688:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077003.753692:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616659712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077003.753715:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616659840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077003.753742:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077003.753757:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077003/real 1769077003] req@ffff995b84643b80 x1855007616659840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077003.753811:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616659840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077003.753823:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077003.753828:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077003.753832:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616659840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077003.753840:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616659968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077003.753864:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077003.753879:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077003/real 1769077003] req@ffff995b84641880 x1855007616659968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077003.753891:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616659968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077003.753901:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077003.753904:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077003.753908:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616659968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077004.456536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077004.456633:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077004.776591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077004.776600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077004.776647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077004.776721:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077005.480563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077005.480681:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077005.800583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077005.800587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077005.800593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077005.800616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077006.504561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077006.504703:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077006.824544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077006.824548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077006.824581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077006.824611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077007.528616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077007.528695:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077007.848556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077007.848618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077007.848630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077007.848689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077008.552635:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077008.552759:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077008.872570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077008.872570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077008.872583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077008.872593:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077008.872600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077008.872604:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077008.872613:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3994 00000100:00100000:1.0:1769077008.872621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077008.872624:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077008.872631:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077008.872663:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616660096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077008.872685:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077008.872688:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077008.872689:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077008.872692:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3994 00000100:00080000:0.0:1769077008.872694:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077008.872698:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077008.872706:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077008.872716:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616660096:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077008.872718:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616660224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077008.872729:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077008.872731:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077008.872733:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077008.872736:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3994 00000100:00080000:0.0:1769077008.872738:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077008.872740:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077008.872743:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077008.872753:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616660352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077008.872759:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077008.872762:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077008.872763:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077008.872766:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3994 00000100:00080000:0.0:1769077008.872769:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077008.872771:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077008.872774:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077008.872778:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077008.872783:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616660480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077008.872791:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077008.872794:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077008.872796:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077008.872798:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3994 00000100:00080000:0.0:1769077008.872801:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077008.872805:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077008.872810:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077008.872823:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616660608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077008.872824:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077008/real 1769077008] req@ffff995b84642d80 x1855007616660096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077008.872859:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077008.872865:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616660096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077008.872866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077008.872871:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077008.872882:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3994 00000100:00080000:3.0:1769077008.872883:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077008.872890:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077008.872891:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077008.872897:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616660096:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077008.872902:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077008.872908:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077008.872918:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616660736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077008.872921:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616660224:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077008.872943:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077008.872946:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077008.872948:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077008.872952:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077008.872953:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3994 00000100:00080000:0.0:1769077008.872959:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077008.872963:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077008.872967:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077008.872976:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077008/real 1769077008] req@ffff995b84643100 x1855007616660224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077008.872977:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616660864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077008.873013:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616660224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077008.873022:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077008.873025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077008.873028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616660224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077008.873032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616660352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077008.873045:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077008.873053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077008/real 1769077008] req@ffff995b84643480 x1855007616660352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077008.873060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616660352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077008.873072:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077008.873074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077008.873076:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616660352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077008.873080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616660480:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077008.873091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077008.873097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077008/real 1769077008] req@ffff995b84641880 x1855007616660480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077008.873104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616660480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077008.873110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077008.873112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077008.873113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616660480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077008.873117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616660608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077008.873134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077008.873160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077008/real 1769077008] req@ffff995b84643b80 x1855007616660608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077008.873169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616660608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077008.873176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077008.873178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077008.873180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616660608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077008.873193:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616660736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077008.873207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077008.873214:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077008/real 1769077008] req@ffff995b84641500 x1855007616660736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077008.873221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616660736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077008.873227:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077008.873228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077008.873231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616660736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077008.873234:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616660864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077008.873243:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077008.873249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077008/real 1769077008] req@ffff995b84641180 x1855007616660864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077008.873256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616660864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077008.873261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077008.873263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077008.873264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616660864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077009.576515:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077009.576616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077009.896535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077009.896539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077009.896540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077009.896570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077010.600534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077010.600614:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077010.920837:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077010.921531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077010.921558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077010.921565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077011.624497:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077011.624577:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077011.944496:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077011.944535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077011.944624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077011.944646:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077012.648518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077012.648569:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077012.968561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077012.968599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077012.968637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077012.968718:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077013.672584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077013.672696:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077013.992560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077013.992583:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077013.992592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077013.992598:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077013.992609:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 3999 00000100:00080000:1.0:1769077013.992622:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077013.992631:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077013.992665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077013.992670:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616660992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077013.992700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077013.992704:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077013.992707:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077013.992713:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3999 00000100:00080000:1.0:1769077013.992716:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077013.992724:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616660992:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077013.992725:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077013.992759:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077013.992782:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616661120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077013.992796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077013.992800:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077013.992803:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077013.992807:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3999 00000100:00080000:1.0:1769077013.992815:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077013.992820:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077013.992826:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077013.992829:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077013.992847:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616661248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077013.992860:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077013.992864:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077013.992867:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077013.992884:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3999 00000100:00100000:3.0:1769077013.992891:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077013/real 1769077013] req@ffff995b868c6300 x1855007616660992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077013.992901:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077013.992915:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077013.992921:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616660992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077013.992928:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769077013.992937:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077013.992945:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077013.992950:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616660992:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077013.992962:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616661376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077013.992979:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077013.992982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616661120:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077013.992986:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077013.992994:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077013.993001:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 3999 00000100:00080000:1.0:1769077013.993008:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077013.993014:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077013.993021:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077013.993036:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077013.993038:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616661504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077013.993076:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077013.993077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077013/real 1769077013] req@ffff995b868c4a80 x1855007616661120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077013.993092:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077013.993116:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077013.993127:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3999 00000100:00100000:3.0:1769077013.993131:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616661120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077013.993136:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077013.993149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077013.993152:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077013.993163:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077013.993167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077013.993175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616661120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077013.993188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616661248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077013.993193:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616661632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077013.993214:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769077013.993220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077013.993226:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077013.993233:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077013.993239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077013/real 1769077013] req@ffff995b868c7800 x1855007616661248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077013.993244:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 3999 00000100:00080000:1.0:1769077013.993255:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077013.993264:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077013.993268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616661248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077013.993273:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077013.993289:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077013.993298:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616661760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077013.993320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077013.993332:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616661248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077013.993358:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616661376:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077013.993392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077013.993406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077013/real 1769077013] req@ffff995b868c6d80 x1855007616661376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077013.993419:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616661376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077013.993429:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077013.993433:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077013.993436:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616661376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077013.993443:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616661504:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077013.993509:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077013.993527:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077013/real 1769077013] req@ffff995b868c5500 x1855007616661504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077013.993540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077013.993545:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616661504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077013.993559:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077013.993564:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077013.993570:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616661504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077013.993578:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616661632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077013.993602:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077013.993605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077013.993616:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077013/real 1769077013] req@ffff995b868c4700 x1855007616661632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077013.993631:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616661632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077013.993643:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077013.993647:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077013.993668:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616661632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077013.993674:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616661760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077013.993691:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077013.993700:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077013/real 1769077013] req@ffff995b868c7480 x1855007616661760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077013.993710:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616661760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077013.993725:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077013.993729:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077013.993732:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616661760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077014.696593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077014.696666:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077015.016545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077015.016592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077015.016618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077015.016636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077015.720569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077015.720723:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077016.040519:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077016.040532:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077016.040561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077016.040607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077016.744561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077016.744661:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077017.064581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077017.064601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077017.064643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077017.064669:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077017.768523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077017.768646:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077018.088636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077018.088676:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077018.088686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077018.088743:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077018.792708:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077018.792790:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077019.112630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077019.112632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077019.112702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077019.112717:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077019.112726:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077019.112732:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077019.112760:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4004 00000100:00100000:3.0:1769077019.112769:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077019.112778:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077019.112792:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077019.112842:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616661888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077019.112898:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077019.112902:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077019.112905:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077019.112909:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4004 00000100:00080000:1.0:1769077019.112915:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077019.112921:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077019.112934:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077019.112946:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616661888:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077019.112955:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616662016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077019.113011:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077019.113015:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077019.113018:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077019.113022:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4004 00000100:00080000:1.0:1769077019.113027:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077019.113032:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077019.113038:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077019.113051:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616662144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077019.113068:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769077019.113072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077019.113074:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077019.113079:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077019.113084:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4004 00000100:00080000:1.0:1769077019.113089:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077019.113094:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077019.113100:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077019.113114:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616662272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077019.113124:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077019.113130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077019/real 1769077019] req@ffff995b868c7100 x1855007616661888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077019.113132:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077019.113137:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077019.113150:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4004 00000100:00080000:1.0:1769077019.113166:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077019.113170:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077019.113175:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077019.113179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616661888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077019.113191:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616662400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077019.113195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077019.113209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077019.113213:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077019.113219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616661888:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077019.113221:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077019.113235:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077019.113247:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4004 00000100:00080000:1.0:1769077019.113251:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077019.113255:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077019.113259:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077019.113270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077019.113272:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616662528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077019.113289:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077019.113292:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077019.113295:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077019.113299:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4004 00000100:00080000:1.0:1769077019.113303:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077019.113313:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769077019.113319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769077019.113323:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077019.113340:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616662656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077019.113342:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077019/real 1769077019] req@ffff995b868c6a00 x1855007616662016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077019.113414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616662016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077019.113430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077019.113446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077019.113477:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077019.113486:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077019.113523:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077019.113536:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077019/real 1769077019] req@ffff995b868c5c00 x1855007616662144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077019.113548:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616662144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077019.113558:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077019.113572:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077019.113575:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077019.113581:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077019.113597:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077019.113605:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077019/real 1769077019] req@ffff995b868c7480 x1855007616662272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077019.113616:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616662272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077019.113631:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077019.113634:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077019.113637:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077019.113642:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077019.113687:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077019.113699:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077019/real 1769077019] req@ffff995b868c4700 x1855007616662400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077019.113712:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616662400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077019.113740:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077019.113743:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077019.113746:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077019.113763:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616662528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077019.113780:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077019.113789:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077019/real 1769077019] req@ffff995b868c5500 x1855007616662528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077019.113801:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616662528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077019.113827:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077019.113831:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077019.113835:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616662528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077019.113842:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616662656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077019.113861:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077019.113873:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077019/real 1769077019] req@ffff995b868c6d80 x1855007616662656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077019.113886:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616662656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077019.113896:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077019.113899:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077019.113903:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616662656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077019.816615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077019.816705:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077020.136603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077020.136644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077020.136676:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077020.136688:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077020.840625:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077020.840767:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077021.160582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077021.160590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077021.160623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077021.160629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077021.864597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077021.864726:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077022.184546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077022.184566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077022.184570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077022.184661:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:2.0:1769077022.888582:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077022.888648:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077023.208573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077023.208574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077023.208574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077023.208684:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077023.912516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077023.912626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077024.232554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077024.232554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077024.232571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077024.232573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077024.232604:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077024.232610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077024.232614:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077024.232621:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4009 00000100:00080000:1.0:1769077024.232631:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077024.232636:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077024.232684:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616662784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077024.232712:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077024.232716:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077024.232718:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077024.232723:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4009 00000100:00080000:1.0:1769077024.232726:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077024.232730:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077024.232749:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077024.232761:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616662912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077024.232766:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616662784:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077024.232774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077024.232778:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077024.232779:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077024.232783:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4009 00000100:00080000:1.0:1769077024.232785:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077024.232788:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077024.232791:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077024.232800:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616663040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077024.232812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077024.232814:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077024.232816:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077024.232819:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4009 00000100:00080000:1.0:1769077024.232822:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077024.232824:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077024.232826:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077024.232836:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616663168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077024.232839:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077024.232848:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077024.232851:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077024.232853:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077024.232857:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4009 00000100:00080000:1.0:1769077024.232861:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077024.232864:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077024.232867:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077024.232874:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616663296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077024.232895:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077024.232899:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077024/real 1769077024] req@ffff995b868c7800 x1855007616662784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077024.232900:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077024.232903:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077024.232910:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4009 00000100:00080000:1.0:1769077024.232917:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077024.232922:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077024.232936:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077024.232945:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616662784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077024.232948:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616663424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077024.232967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077024.232969:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077024.232974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077024.232977:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077024.232978:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077024.232985:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616662784:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077024.232986:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4009 00000100:00080000:1.0:1769077024.232992:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077024.232996:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077024.232999:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077024.233006:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616663552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077024.233017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077024.233047:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077024.233061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077024/real 1769077024] req@ffff995b868c4a80 x1855007616662912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077024.233081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616662912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077024.233087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077024.233090:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077024.233092:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616662912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077024.233097:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077024.233110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077024.233117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077024/real 1769077024] req@ffff995b868c6300 x1855007616663040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077024.233125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616663040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077024.233131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077024.233133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077024.233135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077024.233139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663168:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077024.233149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077024.233165:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077024/real 1769077024] req@ffff995b868c6d80 x1855007616663168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077024.233173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616663168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077024.233179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077024.233181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077024.233183:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077024.233186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663296:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077024.233196:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077024.233202:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077024/real 1769077024] req@ffff995b868c5500 x1855007616663296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077024.233210:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616663296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077024.233215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077024.233217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077024.233219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077024.233222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616663424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077024.233231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077024.233237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077024/real 1769077024] req@ffff995b868c4700 x1855007616663424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077024.233244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616663424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077024.233250:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077024.233252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077024.233254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616663424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077024.233266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616663552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077024.233276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077024.233282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077024/real 1769077024] req@ffff995b868c7480 x1855007616663552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077024.233289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616663552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077024.233295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077024.233297:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077024.233298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616663552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077024.936561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077024.936628:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077025.256538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077025.256565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077025.256569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077025.256581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077025.960539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077025.960610:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077026.280563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077026.280575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077026.280608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077026.280657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077026.984525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077026.984624:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077027.304564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077027.304608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077027.304626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077027.304667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077028.008541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077028.008592:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077028.328587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077028.328591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077028.328591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077028.328592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077029.032550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077029.032651:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077029.352584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077029.352640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077029.352657:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077029.352691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077029.352701:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077029.352706:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077029.352719:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4015 00000100:00080000:3.0:1769077029.352735:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077029.352744:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077029.352746:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077029.352832:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616663680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077029.352866:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077029.352872:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077029.352877:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077029.352884:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4015 00000100:00080000:3.0:1769077029.352890:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077029.352897:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077029.352909:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077029.352915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616663680:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077029.352935:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616663808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077029.352954:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077029.352959:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077029.352963:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077029.352971:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4015 00000100:00080000:3.0:1769077029.352979:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077029.352985:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077029.352992:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077029.353010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077029.353013:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616663936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077029.353030:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077029.353035:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077029.353038:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077029.353046:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4015 00000100:00080000:3.0:1769077029.353051:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077029.353057:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077029.353072:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077029/real 1769077029] req@ffff995b84559500 x1855007616663680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077029.353114:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616663680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077029.353129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077029.353135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077029.353140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616663680:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077029.353172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663808:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769077029.353189:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077029.353215:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616664064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077029.353223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077029.353245:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077029.353253:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077029.353255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077029/real 1769077029] req@ffff995b84558a80 x1855007616663808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077029.353257:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077029.353274:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4015 00000100:00080000:3.0:1769077029.353286:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077029.353296:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077029.353304:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077029.353311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616663808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077029.353324:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616664192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077029.353340:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077029.353344:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077029.353346:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077029.353360:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4015 00000100:00080000:3.0:1769077029.353366:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077029.353369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077029.353372:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077029.353389:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769077029.353390:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077029.353407:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077029.353419:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616664320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077029.353428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663936:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077029.353502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077029.353510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077029.353515:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077029.353526:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4015 00000100:00080000:3.0:1769077029.353535:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077029.353542:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077029.353552:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077029.353565:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077029.353582:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077029/real 1769077029] req@ffff995b8455a680 x1855007616663936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077029.353596:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616664448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077029.353626:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616663936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077029.353666:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077029.353675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077029.353682:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616663936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077029.353715:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664064:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077029.353759:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077029.353781:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077029/real 1769077029] req@ffff995b8455b100 x1855007616664064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077029.353826:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616664064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077029.353838:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077029.353842:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077029.353847:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664064:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077029.353855:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077029.353884:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077029.353899:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077029/real 1769077029] req@ffff995b8455aa00 x1855007616664192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077029.353913:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616664192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077029.353924:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077029.353928:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077029.353933:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077029.353940:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616664320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077029.353960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077029.353971:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077029/real 1769077029] req@ffff995b8455bb80 x1855007616664320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077029.353985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616664320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077029.353996:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077029.354001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077029.354004:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616664320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077029.354013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616664448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077029.354033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077029.354045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077029/real 1769077029] req@ffff995b84558000 x1855007616664448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077029.354062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616664448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077029.354073:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077029.354077:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077029.354083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616664448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077030.056509:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077030.056641:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077030.376581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077030.376581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077030.376581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077030.376584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077031.080530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077031.080619:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077031.400532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077031.400534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077031.400540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077031.400597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077032.104513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077032.104594:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077032.424531:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077032.424538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077032.424545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077032.424563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077033.128535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077033.128649:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077033.448548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077033.448574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077033.448610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077033.448613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077034.152516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077034.152585:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077034.472539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077034.472540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077034.472557:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077034.472563:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077034.472567:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077034.472574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077034.472575:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4020 00000100:00080000:3.0:1769077034.472588:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077034.472594:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077034.472607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077034.472639:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616664576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077034.472661:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077034.472664:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077034.472666:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077034.472670:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4020 00000100:00080000:3.0:1769077034.472673:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077034.472676:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077034.472684:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077034.472705:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616664576:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077034.472706:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616664704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077034.472737:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077034.472741:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077034.472742:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077034.472745:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4020 00000100:00080000:3.0:1769077034.472748:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077034.472751:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077034.472756:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077034.472775:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616664832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077034.472782:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077034.472784:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077034.472786:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077034.472789:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4020 00000100:00080000:3.0:1769077034.472791:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077034.472793:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077034.472796:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077034.472804:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616664960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077034.472811:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077034.472812:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077034.472815:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077034.472818:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077034.472821:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4020 00000100:00080000:3.0:1769077034.472825:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077034.472827:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077034.472831:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077034.472840:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616665088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077034.472847:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077034.472849:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077034.472851:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077034.472854:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077034/real 1769077034] req@ffff995b8455b480 x1855007616664576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077034.472857:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4020 00000100:00080000:3.0:1769077034.472861:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077034.472867:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077034.472872:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077034.472880:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616664576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077034.472882:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616665216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077034.472909:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077034.472910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077034.472917:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077034.472920:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077034.472920:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077034.472928:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616664576:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077034.472929:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4020 00000100:00080000:3.0:1769077034.472934:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077034.472938:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077034.472941:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077034.472948:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616665344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077034.472954:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077034.472974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077034.472985:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077034/real 1769077034] req@ffff995b84558700 x1855007616664704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.472993:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616664704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.472999:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.473001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077034.473004:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077034.473008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077034.473018:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077034.473024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077034/real 1769077034] req@ffff995b8455a300 x1855007616664832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.473032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616664832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.473038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.473040:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077034.473042:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077034.473046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664960:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077034.473056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077034.473063:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077034/real 1769077034] req@ffff995b84559c00 x1855007616664960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.473076:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616664960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.473085:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.473088:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077034.473092:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616664960:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077034.473098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077034.473117:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077034.473129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077034/real 1769077034] req@ffff995b84559880 x1855007616665088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.473138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616665088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.473144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.473146:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077034.473149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077034.473152:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616665216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077034.473165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077034.473171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077034/real 1769077034] req@ffff995b8455ad80 x1855007616665216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.473178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616665216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.473183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.473186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077034.473188:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616665216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077034.473200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616665344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077034.473220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077034.473227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077034/real 1769077034] req@ffff995b84558000 x1855007616665344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.473235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616665344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.473241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.473243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077034.473245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616665344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077035.176610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077035.176692:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077035.496534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077035.496584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077035.496666:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077035.496724:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077036.200539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077036.200587:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077036.520524:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077036.520534:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077036.520547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077036.520578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077037.224521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077037.224567:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077037.544597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077037.544597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077037.544626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077037.544652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077038.248586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077038.248738:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077038.568579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077038.568605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077038.568674:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077038.568725:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077039.272571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077039.272720:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077039.592608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077039.592689:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077039.592697:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077039.592724:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077039.592727:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077039.592735:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077039.592741:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077039.592770:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4025 00000100:00080000:1.0:1769077039.592792:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077039.592804:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077039.592875:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616665472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077039.592925:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077039.592931:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077039.592935:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077039.592942:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4025 00000100:00080000:1.0:1769077039.592951:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077039.592960:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077039.592975:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077039.592979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616665472:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077039.593020:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616665600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077039.593040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077039.593046:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077039.593048:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077039.593053:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4025 00000100:00080000:1.0:1769077039.593067:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077039.593074:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077039.593077:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077039.593086:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077039.593102:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616665728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077039.593112:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077039.593115:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077039.593117:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077039.593126:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4025 00000100:00100000:3.0:1769077039.593128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077039/real 1769077039] req@ffff995b868c5c00 x1855007616665472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077039.593130:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077039.593139:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077039.593148:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077039.593166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616665472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077039.593167:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616665856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077039.593214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077039.593226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077039.593233:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077039.593234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616665472:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077039.593248:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077039.593254:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077039.593262:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4025 00000100:00080000:1.0:1769077039.593269:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077039.593272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665600:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077039.593281:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077039.593309:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077039.593330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077039.593334:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616665984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077039.593350:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077039.593353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077039/real 1769077039] req@ffff995b868c6a00 x1855007616665600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077039.593357:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077039.593365:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077039.593378:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4025 00000100:00080000:1.0:1769077039.593388:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077039.593397:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077039.593400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616665600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077039.593403:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077039.593419:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616666112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077039.593430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077039.593435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077039.593458:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077039.593463:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077039.593464:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077039.593474:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4025 00000100:00080000:1.0:1769077039.593480:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077039.593482:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665600:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077039.593515:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077039.593522:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077039.593528:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077039.593536:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616666240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077039.593568:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077039.593584:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077039/real 1769077039] req@ffff995b868c7100 x1855007616665728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077039.593596:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616665728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077039.593606:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077039.593610:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077039.593615:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077039.593642:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665856:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077039.593695:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077039.593716:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077039/real 1769077039] req@ffff995b868c7480 x1855007616665856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077039.593738:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616665856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077039.593753:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077039.593757:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077039.593762:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077039.593782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665984:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077039.593825:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077039.593837:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077039/real 1769077039] req@ffff995b868c4700 x1855007616665984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077039.593848:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616665984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077039.593857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077039.593860:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077039.593863:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616665984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077039.593868:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616666112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077039.593883:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077039.593892:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077039/real 1769077039] req@ffff995b868c5500 x1855007616666112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077039.593902:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616666112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077039.593910:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077039.593912:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077039.593915:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616666112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077039.593920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616666240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077039.593933:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077039.593941:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077039/real 1769077039] req@ffff995b868c6d80 x1855007616666240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077039.593952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616666240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077039.593960:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077039.593962:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077039.593965:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616666240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077040.296563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077040.296656:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077040.616557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077040.616563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077040.616576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077040.616641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077041.320566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077041.320689:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077041.640731:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077041.640761:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077041.642282:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077041.642387:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077042.344629:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077042.344770:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077042.664557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077042.664557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077042.664557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077042.664634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077043.368555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077043.368688:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077043.688595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077043.688637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077043.688645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077043.688665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077044.392565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077044.392657:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077044.712617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077044.712618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077044.712636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077044.712641:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077044.712649:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077044.712653:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077044.712663:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4030 00000100:00100000:0.0:1769077044.712673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077044.712675:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077044.712730:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077044.712767:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616666368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077044.712789:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077044.712792:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077044.712794:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077044.712798:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4030 00000100:00080000:1.0:1769077044.712801:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077044.712805:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077044.712813:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077044.712842:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616666496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077044.712843:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616666368:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077044.712857:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077044.712860:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077044.712861:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077044.712864:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4030 00000100:00080000:1.0:1769077044.712867:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077044.712869:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077044.712873:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077044.712881:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616666624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077044.712888:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077044.712890:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077044.712891:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077044.712893:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4030 00000100:00080000:1.0:1769077044.712896:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077044.712899:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077044.712903:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077044.712914:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077044.712915:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616666752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077044.712924:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077044.712928:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077044.712930:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077044.712934:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4030 00000100:00080000:1.0:1769077044.712938:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077044.712943:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077044.712948:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077044.712951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077044/real 1769077044] req@ffff995b868c6300 x1855007616666368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077044.712982:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616666880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077044.712988:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616666368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077044.713015:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077044.713017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077044.713038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077044.713039:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077044.713043:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077044.713044:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616666368:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077044.713050:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4030 00000100:00080000:1.0:1769077044.713054:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077044.713058:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077044.713062:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077044.713070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616666496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077044.713075:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616667008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077044.713085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077044.713088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077044.713090:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077044.713093:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4030 00000100:00080000:1.0:1769077044.713097:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077044.713099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077044.713102:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077044.713109:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077044.713119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077044/real 1769077044] req@ffff995b868c4a80 x1855007616666496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077044.713130:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616667136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077044.713165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616666496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077044.713173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077044.713176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077044.713179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616666496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077044.713183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616666624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077044.713209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077044.713217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077044/real 1769077044] req@ffff995b868c7800 x1855007616666624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077044.713225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616666624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077044.713233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077044.713235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077044.713237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616666624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077044.713242:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616666752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077044.713252:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077044.713258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077044/real 1769077044] req@ffff995b868c6d80 x1855007616666752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077044.713270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616666752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077044.713276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077044.713278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077044.713280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616666752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077044.713283:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616666880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077044.713292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077044.713298:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077044/real 1769077044] req@ffff995b868c5500 x1855007616666880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077044.713305:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616666880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077044.713311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077044.713313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077044.713314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616666880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077044.713326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616667008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077044.713336:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077044.713342:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077044/real 1769077044] req@ffff995b868c4700 x1855007616667008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077044.713350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616667008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077044.713360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077044.713362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077044.713364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616667008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077044.713367:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616667136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077044.713376:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077044.713382:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077044/real 1769077044] req@ffff995b868c7480 x1855007616667136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077044.713398:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616667136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077044.713403:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077044.713405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077044.713406:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616667136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077045.416553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077045.416634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077045.736565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077045.736565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077045.736580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077045.736603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077046.440629:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077046.440744:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077046.760541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077046.760542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077046.760559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077046.760590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077047.464598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077047.464656:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077047.784578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077047.784616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077047.784623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077047.784653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077048.488557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077048.488629:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077048.808619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077048.808628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077048.808683:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077048.808690:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077049.512562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077049.512586:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077049.832620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077049.832659:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077049.832667:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077049.832671:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077049.832681:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077049.832689:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077049.832720:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4035 00000100:00080000:1.0:1769077049.832758:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077049.832772:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077049.832784:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077049.832852:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616667264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077049.832899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077049.832905:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077049.832907:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077049.832912:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4035 00000100:00080000:1.0:1769077049.832918:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077049.832924:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077049.832936:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077049.832955:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616667392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077049.832969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077049.832970:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616667264:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077049.832979:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077049.832984:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077049.832990:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4035 00000100:00080000:1.0:1769077049.832994:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077049.832997:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077049.833003:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077049.833016:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616667520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077049.833027:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077049.833032:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077049.833034:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077049.833038:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4035 00000100:00080000:1.0:1769077049.833041:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077049.833046:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077049.833051:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077049.833069:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616667648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077049.833070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077049.833085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077049.833090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077049.833093:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077049.833097:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4035 00000100:00080000:1.0:1769077049.833102:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077049.833106:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077049.833111:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077049.833125:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616667776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077049.833135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077049/real 1769077049] req@ffff995b868c7100 x1855007616667264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077049.833138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077049.833150:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077049.833157:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077049.833175:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4035 00000100:00080000:1.0:1769077049.833186:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077049.833195:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616667264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077049.833196:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077049.833210:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077049.833229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077049.833242:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616667904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077049.833263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077049.833268:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077049.833278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616667264:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077049.833281:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077049.833289:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077049.833301:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4035 00000100:00080000:1.0:1769077049.833309:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077049.833316:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077049.833326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616667392:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769077049.833329:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077049.833358:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616668032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077049.833386:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077049.833418:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077049/real 1769077049] req@ffff995b868c6a00 x1855007616667392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077049.833443:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616667392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077049.833535:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077049.833542:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077049.833549:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616667392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077049.833560:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616667520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077049.833601:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077049.833623:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077049/real 1769077049] req@ffff995b868c5c00 x1855007616667520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077049.833647:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616667520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077049.833695:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077049.833702:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077049.833709:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616667520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077049.833721:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616667648:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077049.833754:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077049.833769:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077049/real 1769077049] req@ffff995b868c7480 x1855007616667648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077049.833784:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616667648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077049.833796:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077049.833800:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077049.833804:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616667648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077049.833811:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616667776:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077049.833830:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077049.833859:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077049/real 1769077049] req@ffff995b868c4700 x1855007616667776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077049.833871:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616667776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077049.833880:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077049.833883:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077049.833887:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616667776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077049.833893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616667904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077049.833909:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077049.833919:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077049/real 1769077049] req@ffff995b868c5500 x1855007616667904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077049.833930:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616667904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077049.833939:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077049.833942:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077049.833945:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616667904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077049.833966:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616668032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077049.833983:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077049.833993:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077049/real 1769077049] req@ffff995b868c6d80 x1855007616668032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077049.834006:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616668032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077049.834015:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077049.834017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077049.834021:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616668032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077050.536526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077050.536599:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077050.856603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077050.856616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077050.856645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077050.856646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077051.560556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077051.560606:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077051.880589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077051.880589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077051.880626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077051.880657:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077052.584518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077052.584634:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077052.904572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077052.904572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077052.904572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077052.904574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077053.608526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077053.608629:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077053.928552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077053.928569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077053.928593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077053.928625:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077054.632527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077054.632606:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077054.952548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077054.952548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077054.952558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077054.952589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077054.952589:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077054.952596:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077054.952600:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077054.952608:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4040 00000100:00080000:2.0:1769077054.952618:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077054.952624:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077054.952670:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616668160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077054.952691:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077054.952694:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077054.952695:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077054.952699:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4040 00000100:00080000:2.0:1769077054.952712:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077054.952716:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077054.952728:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077054.952734:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616668160:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077054.952742:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616668288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077054.952750:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077054.952753:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077054.952755:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077054.952757:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4040 00000100:00080000:2.0:1769077054.952760:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077054.952762:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077054.952765:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077054.952788:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616668416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077054.952795:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077054.952797:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077054.952799:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077054.952802:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4040 00000100:00080000:2.0:1769077054.952804:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077054.952806:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077054.952809:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077054.952814:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077054.952819:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616668544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077054.952827:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077054.952830:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077054.952832:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077054.952845:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4040 00000100:00080000:2.0:1769077054.952847:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077054.952850:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077054.952853:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077054.952862:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616668672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077054.952865:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077054/real 1769077054] req@ffff995b85ac8380 x1855007616668160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077054.952883:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077054.952886:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077054.952887:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077054.952890:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4040 00000100:00080000:2.0:1769077054.952893:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077054.952896:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077054.952909:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077054.952916:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616668800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077054.952921:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077054.952923:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077054.952924:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077054.952927:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4040 00000100:00080000:2.0:1769077054.952929:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077054.952931:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077054.952934:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077054.952939:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616668928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077054.952961:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616668160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077054.952986:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077054.952990:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077054.952993:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616668160:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077054.953015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616668288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077054.953047:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077054.953061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077054/real 1769077054] req@ffff995b85acbb80 x1855007616668288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077054.953076:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616668288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077054.953087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077054.953091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077054.953095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616668288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077054.953101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616668416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077054.953117:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077054.953125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077054/real 1769077054] req@ffff995b85ac9880 x1855007616668416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077054.953134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616668416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077054.953142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077054.953144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077054.953146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616668416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077054.953150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616668544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077054.953161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077054.953175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077054/real 1769077054] req@ffff995b85ac9c00 x1855007616668544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077054.953183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616668544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077054.953188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077054.953190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077054.953192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616668544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077054.953196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616668672:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077054.953205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077054.953211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077054/real 1769077054] req@ffff995b85acb800 x1855007616668672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077054.953218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616668672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077054.953229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077054.953231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077054.953233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616668672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077054.953236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616668800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077054.953254:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077054.953260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077054/real 1769077054] req@ffff995b85aca300 x1855007616668800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077054.953268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616668800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077054.953273:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077054.953275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077054.953277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616668800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077054.953280:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616668928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077054.953289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077054.953295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077054/real 1769077054] req@ffff995b85acaa00 x1855007616668928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077054.953302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616668928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077054.953306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077054.953308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077054.953311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616668928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077055.656545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077055.656633:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077055.976588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077055.976636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077055.976691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077055.976751:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077056.680581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077056.680694:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077057.000540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077057.000546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077057.000553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077057.000577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077057.704547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077057.704631:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077058.024555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077058.024596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077058.024615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077058.024653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077058.728537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077058.728724:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077059.048568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077059.048571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077059.048648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077059.048672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077059.752636:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077059.752716:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077060.072605:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077060.072614:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769077060.072618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077060.072618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077060.072621:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077060.072640:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4045 00000100:00080000:0.0:1769077060.072654:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077060.072661:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077060.072663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077060.072679:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077060.072702:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616669056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077060.072726:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077060.072731:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077060.072733:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077060.072739:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4045 00000100:00080000:0.0:1769077060.072742:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077060.072746:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077060.072756:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077060.072764:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616669056:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077060.072770:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616669184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077060.072780:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077060.072783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077060.072785:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077060.072789:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4045 00000100:00080000:0.0:1769077060.072795:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077060.072797:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077060.072800:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077060.072811:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616669312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077060.072817:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077060.072821:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077060.072822:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077060.072824:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4045 00000100:00080000:0.0:1769077060.072827:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077060.072829:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077060.072831:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077060.072840:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616669440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077060.072846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769077060.072853:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077060.072854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077060.072878:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077060.072885:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4045 00000100:00080000:0.0:1769077060.072893:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077060.072897:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077060.072914:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077060.072926:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616669568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077060.072950:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077060.072953:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077060.072955:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077060.072963:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4045 00000100:00100000:3.0:1769077060.072963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077060/real 1769077060] req@ffff995b84642a00 x1855007616669056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077060.072972:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077060.072982:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077060.073009:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077060.073031:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616669056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077060.073032:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616669696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077060.073078:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077060.073086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077060.073093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077060.073098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077060.073102:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077060.073106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616669056:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077060.073115:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4045 00000100:00080000:0.0:1769077060.073120:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077060.073124:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077060.073131:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077060.073141:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616669824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077060.073143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616669184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077060.073204:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077060.073216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077060/real 1769077060] req@ffff995b84643800 x1855007616669184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077060.073225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616669184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077060.073233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077060.073237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077060.073239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616669184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077060.073245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616669312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077060.073258:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077060.073265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077060/real 1769077060] req@ffff995b84640700 x1855007616669312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077060.073273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616669312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077060.073305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077060.073308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077060.073311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616669312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077060.073316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616669440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077060.073333:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077060.073341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077060/real 1769077060] req@ffff995b84642680 x1855007616669440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077060.073350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616669440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077060.073357:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077060.073359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077060.073362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616669440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077060.073366:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616669568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077060.073378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077060.073384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077060/real 1769077060] req@ffff995b84641c00 x1855007616669568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077060.073393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616669568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077060.073399:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077060.073401:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077060.073403:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616669568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077060.073407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616669696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077060.073418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077060.073424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077060/real 1769077060] req@ffff995b84640a80 x1855007616669696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077060.073432:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616669696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077060.073438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077060.073440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077060.073442:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616669696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077060.073481:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616669824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077060.073494:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077060.073502:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077060/real 1769077060] req@ffff995b84640000 x1855007616669824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077060.073517:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616669824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077060.073544:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077060.073546:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077060.073549:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616669824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077060.776709:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077060.776745:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077061.096551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077061.096551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077061.096556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077061.096587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077061.800587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077061.800664:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077062.120576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077062.120576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077062.120576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077062.120613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077062.824571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077062.824663:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077063.144590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077063.144591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077063.144597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077063.144633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077063.848635:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077063.848757:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077064.168567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077064.168629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077064.169562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077064.169580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077064.872623:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077064.872722:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077065.192637:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077065.192637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077065.192667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077065.192704:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077065.192713:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769077065.192714:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077065.192719:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077065.192743:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4050 00000100:00080000:1.0:1769077065.192755:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077065.192762:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077065.192803:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616669952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077065.192827:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077065.192831:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077065.192833:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077065.192838:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4050 00000100:00080000:1.0:1769077065.192842:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077065.192846:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077065.192856:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077065.192866:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616670080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077065.192875:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077065.192878:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077065.192879:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077065.192883:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4050 00000100:00080000:1.0:1769077065.192886:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077065.192889:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077065.192894:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077065.192903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616669952:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077065.192904:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616670208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077065.192931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077065.192936:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077065.192940:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077065.192948:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4050 00000100:00080000:1.0:1769077065.192955:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077065.192962:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077065.192970:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077065.193012:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616670336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077065.193030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077065.193035:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077065.193038:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077065.193044:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4050 00000100:00080000:1.0:1769077065.193069:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077065.193070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670080:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077065.193090:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077065.193103:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077065.193144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077065.193150:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616670464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077065.193168:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670336:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769077065.193171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077065.193178:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077065.193183:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077065.193191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670464:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769077065.193211:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4050 00000100:00080000:1.0:1769077065.193221:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077065.193229:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077065.193240:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077065.193264:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616670592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077065.193295:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077065.193302:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077065.193306:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077065.193317:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4050 00000100:00080000:1.0:1769077065.193328:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077065.193337:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077065.193340:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616670592:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769077065.193359:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077065.193385:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616670720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077065.193436:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616670720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077065.896559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077065.896620:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077066.216578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077066.216578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077066.216624:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077066.216641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077066.920533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077066.920615:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077067.240567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077067.240567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077067.240571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077067.240605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077067.944537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077067.944596:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077068.264535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077068.264549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077068.264555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077068.264684:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:1.0:1769077068.264727:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769077068.264765:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077068.264782:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077068.264812:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077068.264821:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077068.264830:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077068.264835:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077068.264840:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769077068.264846:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077065/real 1769077068] req@ffff995b868c6300 x1855007616670208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1769077068.264847:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769077068.264872:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616670208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077068.264885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077068.264889:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077068.264893:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077068.264898:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077065/real 1769077068] req@ffff995b868c6d80 x1855007616670336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077068.264906:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616670336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077068.264913:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077068.264915:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077068.264917:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077068.264929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077065/real 1769077068] req@ffff995b868c7800 x1855007616669952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077068.264937:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616669952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077068.264942:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077068.264944:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077068.264946:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616669952:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077068.264951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077065/real 1769077068] req@ffff995b868c4a80 x1855007616670080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077068.264957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616670080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077068.264964:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077068.264966:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077068.264968:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077068.264991:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077065/real 1769077068] req@ffff995b868c7480 x1855007616670720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077068.264998:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616670720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077068.265004:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077068.265007:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077068.265009:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616670720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077068.265013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077065/real 1769077068] req@ffff995b868c4700 x1855007616670592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077068.265020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616670592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077068.265025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077068.265028:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077068.265031:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616670592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077068.265037:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077065/real 1769077068] req@ffff995b868c5500 x1855007616670464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077068.265048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616670464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077068.265059:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077068.265062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077068.265066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077068.968577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077068.968678:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077069.288552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077069.288557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077069.288558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077069.288594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077069.992528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077069.992618:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077070.312561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077070.312571:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077070.312572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077070.312582:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077070.312588:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077070.312599:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4056 00000100:00100000:2.0:1769077070.312610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077070.312611:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077070.312623:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077070.312625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077070.312667:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616670848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077070.312694:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077070.312698:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077070.312700:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077070.312704:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4056 00000100:00080000:3.0:1769077070.312708:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077070.312712:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077070.312722:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077070.312738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616670848:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077070.312739:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616670976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077070.312755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077070.312758:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077070.312759:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077070.312763:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4056 00000100:00080000:3.0:1769077070.312765:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077070.312768:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077070.312772:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077070.312781:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616671104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077070.312789:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077070.312793:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077070.312795:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077070.312816:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4056 00000100:00080000:3.0:1769077070.312821:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077070.312826:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077070.312832:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769077070.312849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077070.312850:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616671232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077070.312866:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077070.312871:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077070.312874:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077070.312880:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4056 00000100:00080000:3.0:1769077070.312884:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077070.312889:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077070.312897:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077070.312907:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077070/real 1769077070] req@ffff995b8455bb80 x1855007616670848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077070.312916:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616671360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077070.312938:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077070.312945:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769077070.312946:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616670848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077070.312948:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077070.312961:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4056 00000100:00080000:0.0:1769077070.312971:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077070.312971:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077070.312981:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769077070.312984:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769077070.312992:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077070.312993:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616670848:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077070.313013:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616671488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077070.313034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670976:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077070.313039:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077070.313049:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077070.313052:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077070.313061:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4056 00000100:00080000:3.0:1769077070.313067:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077070.313070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077070.313077:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077070.313085:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077070.313094:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077070/real 1769077070] req@ffff995b8455aa00 x1855007616670976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077070.313103:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616671616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077070.313152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616670976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077070.313171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077070.313175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077070.313180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616670976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077070.313188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616671104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077070.313213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077070.313228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077070/real 1769077070] req@ffff995b8455b100 x1855007616671104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077070.313241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616671104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077070.313254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077070.313258:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077070.313262:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616671104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077070.313269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616671232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077070.313292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077070.313305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077070/real 1769077070] req@ffff995b8455a680 x1855007616671232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077070.313319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616671232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077070.313338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077070.313342:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077070.313347:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616671232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077070.313355:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616671360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077070.313380:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077070.313395:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077070/real 1769077070] req@ffff995b84558a80 x1855007616671360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077070.313410:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616671360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077070.313422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077070.313426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077070.313430:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616671360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077070.313480:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616671488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077070.313514:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077070.313526:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077070/real 1769077070] req@ffff995b84559500 x1855007616671488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077070.313539:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616671488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077070.313551:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077070.313554:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077070.313558:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616671488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077070.313582:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616671616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077070.313604:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077070.313616:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077070/real 1769077070] req@ffff995b8455bb80 x1855007616671616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077070.313629:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616671616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077070.313638:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077070.313641:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077070.313645:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616671616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077071.016545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077071.016618:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077071.336540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077071.336546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077071.336573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077071.336623:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077072.040542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077072.040587:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077072.360537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077072.360585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077072.360598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077072.360598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077073.064527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077073.064614:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077073.384563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077073.384563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077073.384582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077073.384618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077074.088532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077074.088589:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077074.408550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077074.408550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077074.408551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077074.408562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077075.112579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077075.112622:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077075.432578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077075.432578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077075.432587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077075.432626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077075.432643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077075.432649:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077075.432654:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077075.432663:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4061 00000100:00080000:2.0:1769077075.432674:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077075.432681:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077075.432728:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616671744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077075.432755:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077075.432760:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077075.432761:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077075.432764:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4061 00000100:00080000:2.0:1769077075.432767:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077075.432769:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077075.432778:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077075.432788:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616671872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077075.432797:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077075.432799:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616671744:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077075.432800:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077075.432802:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077075.432810:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4061 00000100:00080000:2.0:1769077075.432813:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077075.432815:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077075.432818:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077075.432829:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616672000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077075.432835:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077075.432837:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077075.432839:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077075.432842:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4061 00000100:00080000:2.0:1769077075.432844:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077075.432846:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077075.432851:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077075.432858:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616672128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077075.432865:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077075.432867:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077075.432868:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769077075.432873:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077075.432874:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4061 00000100:00080000:2.0:1769077075.432880:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077075.432882:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077075.432886:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077075.432895:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616672256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077075.432902:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077075.432904:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077075.432905:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077075.432909:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4061 00000100:00080000:2.0:1769077075.432912:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077075.432917:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077075/real 1769077075] req@ffff995b85acb100 x1855007616671744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077075.432917:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077075.432930:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077075.432936:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616671744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077075.432942:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616672384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077075.432948:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077075.432952:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077075.432956:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077075.432957:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077075.432961:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077075.432963:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616671744:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077075.432968:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4061 00000100:00080000:2.0:1769077075.432973:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077075.432976:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077075.432979:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077075.432986:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616672512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077075.432992:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616671872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077075.433014:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077075.433043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077075/real 1769077075] req@ffff995b85ac8a80 x1855007616671872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077075.433053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616671872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077075.433059:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077075.433062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077075.433064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616671872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077075.433068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077075.433083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077075.433090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077075/real 1769077075] req@ffff995b85acaa00 x1855007616672000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077075.433097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616672000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077075.433104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077075.433106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077075.433108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077075.433112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672128:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077075.433122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077075.433127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077075/real 1769077075] req@ffff995b85aca300 x1855007616672128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077075.433139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616672128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077075.433145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077075.433147:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077075.433149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077075.433152:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077075.433161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077075.433167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077075/real 1769077075] req@ffff995b85acb800 x1855007616672256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077075.433174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616672256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077075.433180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077075.433182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077075.433184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077075.433187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616672384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077075.433196:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077075.433201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077075/real 1769077075] req@ffff995b85ac9c00 x1855007616672384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077075.433209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616672384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077075.433219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077075.433221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077075.433223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616672384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077075.433236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616672512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077075.433246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077075.433252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077075/real 1769077075] req@ffff995b85ac9880 x1855007616672512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077075.433259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616672512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077075.433265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077075.433267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077075.433276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616672512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077076.136544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077076.136623:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077076.456584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077076.456584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077076.456611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077076.456639:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077077.160601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077077.160691:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077077.480550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077077.480584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077077.480601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077077.480628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077078.184582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077078.184671:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077078.504537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077078.504537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077078.504542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077078.504603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769077079.208600:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077079.208606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077079.528533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077079.528536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077079.528572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077079.528604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077080.232536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077080.232620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077080.552536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077080.552555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077080.552567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077080.552623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077080.552631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077080.552635:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077080.552647:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4066 00000100:00100000:2.0:1769077080.552652:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077080.552658:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077080.552666:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077080.552710:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616672640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077080.552743:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077080.552747:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077080.552749:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077080.552761:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4066 00000100:00080000:0.0:1769077080.552766:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077080.552773:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077080.552774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616672640:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077080.552785:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077080.552804:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616672768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077080.552818:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077080.552822:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077080.552824:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077080.552829:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4066 00000100:00080000:0.0:1769077080.552834:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077080.552839:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077080.552840:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077080.552849:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077080.552865:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616672896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077080.552876:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077080.552877:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077080/real 1769077080] req@ffff995b84640e00 x1855007616672640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077080.552887:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077080.552892:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077080.552900:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4066 00000100:00100000:3.0:1769077080.552902:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616672640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077080.552908:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077080.552915:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077080.552918:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077080.552922:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077080.552928:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616672640:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077080.552929:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077080.552965:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616673024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077080.552971:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672768:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077080.552988:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077080.552993:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077080.552996:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077080.552999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077080.553007:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4066 00000100:00080000:0.0:1769077080.553014:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077080.553014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077080/real 1769077080] req@ffff995b84640380 x1855007616672768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077080.553023:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077080.553032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616672768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077080.553033:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769077080.553046:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077080.553050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077080.553056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077080.553057:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616673152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077080.553069:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672896:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077080.553073:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077080.553079:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077080.553081:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077080.553086:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4066 00000100:00080000:0.0:1769077080.553091:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077080.553096:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769077080.553102:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077080.553105:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077080.553119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077080/real 1769077080] req@ffff995b84641f80 x1855007616672896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077080.553124:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616673280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077080.553135:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616672896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077080.553144:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077080.553149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077080.553150:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077080.553155:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077080.553157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077080.553167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616672896:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077080.553168:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4066 00000100:00080000:0.0:1769077080.553175:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077080.553181:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077080.553185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673024:192.168.204.132@tcp:8:kworker.0 00000008:00100000:0.0:1769077080.553189:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077080.553207:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616673408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077080.553208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077080.553219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077080/real 1769077080] req@ffff995b84640000 x1855007616673024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077080.553232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616673024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077080.553238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077080.553240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077080.553243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077080.553248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673152:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077080.553267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077080.553279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077080/real 1769077080] req@ffff995b84640a80 x1855007616673152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077080.553292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616673152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077080.553302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077080.553306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077080.553309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077080.553316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616673280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077080.553336:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077080.553347:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077080/real 1769077080] req@ffff995b84641c00 x1855007616673280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077080.553360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616673280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077080.553376:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077080.553379:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077080.553383:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616673280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077080.553396:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616673408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077080.553421:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077080.553430:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077080/real 1769077080] req@ffff995b84642680 x1855007616673408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077080.553438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616673408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077080.553443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077080.553446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077080.553448:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616673408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077081.256551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077081.256618:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077081.576535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077081.576541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077081.576572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077081.576584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077082.280544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077082.280606:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077082.600513:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077082.600532:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077082.600590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077082.600592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077083.304549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077083.304620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077083.624540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077083.624549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077083.624554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077083.624559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077084.328557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077084.328639:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077084.648551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077084.648552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077084.648578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077084.648582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077085.352559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077085.352659:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077085.672597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077085.672606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077085.672607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077085.672613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077085.672623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077085.672631:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077085.672644:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4071 00000100:00080000:0.0:1769077085.672659:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077085.672671:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077085.672671:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077085.672717:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616673536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077085.672742:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077085.672746:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077085.672748:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077085.672751:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4071 00000100:00080000:0.0:1769077085.672756:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077085.672760:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077085.672770:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077085.672782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616673536:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077085.672784:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616673664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077085.672804:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077085.672806:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077085.672807:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077085.672810:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4071 00000100:00080000:0.0:1769077085.672814:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077085.672818:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077085.672822:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077085.672837:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616673792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077085.672846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077085.672877:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077085.672880:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077085.672885:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4071 00000100:00080000:0.0:1769077085.672892:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077085.672892:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077085.672903:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077085.672911:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077085.672926:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616673920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077085.672936:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077085.672939:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077085.672941:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077085.672947:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077085/real 1769077085] req@ffff995b84640700 x1855007616673536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077085.672949:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4071 00000100:00080000:0.0:1769077085.672957:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077085.672971:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077085.672980:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077085.672986:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616673536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077085.672993:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616674048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077085.673022:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077085.673025:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077085.673027:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077085.673027:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077085.673038:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4071 00000100:00080000:0.0:1769077085.673045:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077085.673045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077085.673053:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077085.673056:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077085.673059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616673536:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077085.673067:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616674176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077085.673074:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077085.673077:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077085.673079:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077085.673085:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4071 00000100:00080000:0.0:1769077085.673088:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077085.673091:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673664:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077085.673093:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077085.673106:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077085.673119:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616674304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077085.673144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077085.673161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077085/real 1769077085] req@ffff995b84643800 x1855007616673664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077085.673173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616673664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077085.673182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077085.673186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077085.673190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077085.673198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077085.673219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077085.673231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077085/real 1769077085] req@ffff995b84642a00 x1855007616673792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077085.673245:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616673792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077085.673253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077085.673256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077085.673259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077085.673264:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077085.673280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077085.673287:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077085/real 1769077085] req@ffff995b84641180 x1855007616673920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077085.673295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616673920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077085.673302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077085.673304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077085.673306:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616673920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077085.673310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077085.673320:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077085.673326:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077085/real 1769077085] req@ffff995b84641500 x1855007616674048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077085.673333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616674048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077085.673339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077085.673341:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077085.673343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077085.673347:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616674176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077085.673358:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077085.673364:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077085/real 1769077085] req@ffff995b84643b80 x1855007616674176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077085.673372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616674176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077085.673377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077085.673379:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077085.673381:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616674176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077085.673401:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616674304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077085.673422:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077085.673435:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077085/real 1769077085] req@ffff995b84641880 x1855007616674304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077085.673448:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616674304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077085.673476:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077085.673479:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077085.673482:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616674304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077086.376559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077086.376683:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077086.696631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077086.696684:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077086.696714:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077086.696732:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077087.400567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077087.400605:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077087.720576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077087.720576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077087.720612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077087.720622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077088.424565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077088.424658:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077088.744564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077088.744584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077088.744616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077088.744627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077089.448568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077089.448698:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077089.768567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077089.768567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077089.768567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077089.768577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077090.472533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077090.472642:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077090.792561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077090.792577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077090.792602:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077090.792606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077090.792615:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077090.792621:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077090.792628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077090.792635:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4076 00000100:00080000:3.0:1769077090.792647:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077090.792656:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077090.792702:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616674432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077090.792737:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077090.792746:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077090.792747:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077090.792751:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4076 00000100:00080000:3.0:1769077090.792756:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077090.792760:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077090.792781:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077090.792796:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616674560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077090.792814:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616674432:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077090.792841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077090.792845:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077090.792847:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077090.792854:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4076 00000100:00080000:3.0:1769077090.792857:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077090.792861:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077090.792865:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077090.792879:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616674688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077090.792886:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077090.792889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077090.792891:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077090.792894:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4076 00000100:00080000:3.0:1769077090.792897:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077090.792900:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077090.792904:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077090.792911:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616674816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077090.792922:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077090.792924:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077090.792930:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077090.792932:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077090.792935:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4076 00000100:00080000:3.0:1769077090.792938:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077090.792941:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077090.792946:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077090.792958:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616674944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077090.792966:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077090.792969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077090.792971:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077090.792977:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4076 00000100:00080000:3.0:1769077090.792982:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077090.792987:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077090.792993:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077090/real 1769077090] req@ffff995b84558000 x1855007616674432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077090.792993:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077090.793018:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616675072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077090.793024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616674432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077090.793058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077090.793064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077090.793065:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077090.793069:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4076 00000100:00080000:3.0:1769077090.793074:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077090.793076:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077090.793079:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769077090.793086:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077090.793104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616674432:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769077090.793105:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077090.793129:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616675200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077090.793147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077090.793182:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077090.793194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077090/real 1769077090] req@ffff995b8455ad80 x1855007616674560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077090.793208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616674560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077090.793215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077090.793218:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077090.793220:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077090.793224:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077090.793237:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077090.793244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077090/real 1769077090] req@ffff995b84559880 x1855007616674688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077090.793252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616674688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077090.793259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077090.793261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077090.793263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077090.793267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674816:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077090.793278:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077090.793284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077090/real 1769077090] req@ffff995b84559c00 x1855007616674816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077090.793291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616674816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077090.793303:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077090.793305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077090.793307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077090.793310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674944:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077090.793320:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077090.793326:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077090/real 1769077090] req@ffff995b8455a300 x1855007616674944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077090.793332:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616674944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077090.793338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077090.793340:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077090.793342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616674944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077090.793346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616675072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077090.793355:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077090.793362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077090/real 1769077090] req@ffff995b84558700 x1855007616675072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077090.793368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616675072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077090.793374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077090.793376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077090.793379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616675072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077090.793393:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616675200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077090.793403:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077090.793409:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077090/real 1769077090] req@ffff995b8455b480 x1855007616675200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077090.793416:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616675200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077090.793421:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077090.793424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077090.793426:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616675200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077091.496579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077091.496640:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077091.816623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077091.816634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077091.816634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077091.816656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077092.520524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077092.520613:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077092.840547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077092.840547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077092.840591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077092.840593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077093.544580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077093.544621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077093.864533:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077093.864563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077093.864597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077093.864629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077094.568553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077094.568613:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077094.888588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077094.888651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077094.888661:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077094.888682:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077095.592570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077095.592637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077095.912596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077095.912596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077095.912616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077095.912629:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077095.912637:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077095.912645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077095.912653:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4081 00000100:00080000:2.0:1769077095.912670:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077095.912674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077095.912717:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077095.912760:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616675328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077095.912792:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077095.912796:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077095.912799:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077095.912806:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4081 00000100:00080000:2.0:1769077095.912811:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077095.912815:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077095.912824:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616675328:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077095.912825:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077095.912847:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616675456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077095.912861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077095.912863:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077095.912865:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077095.912868:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4081 00000100:00080000:2.0:1769077095.912870:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077095.912891:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077095.912896:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077095.912909:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616675584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077095.912918:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077095.912921:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077095.912923:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769077095.912931:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077095.912936:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4081 00000100:00080000:2.0:1769077095.912961:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077095.912965:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077095.912970:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077095.912982:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616675712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077095.912989:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077095.912992:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077095.912993:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077095.912996:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4081 00000100:00080000:2.0:1769077095.912999:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077095.913001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077095/real 1769077095] req@ffff995b85acbb80 x1855007616675328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077095.913005:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077095.913014:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077095.913052:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616675840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077095.913055:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616675328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077095.913102:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077095.913103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077095.913112:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077095.913118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077095.913118:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077095.913140:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4081 00000100:00100000:0.0:1769077095.913141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616675328:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077095.913152:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077095.913157:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077095.913162:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077095.913171:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616675968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077095.913178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616675456:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077095.913194:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077095.913198:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077095.913201:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077095.913209:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4081 00000100:00080000:2.0:1769077095.913213:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077095.913219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077095.913225:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077095.913232:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077095.913239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077095/real 1769077095] req@ffff995b85ac8380 x1855007616675456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077095.913245:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616676096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077095.913269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616675456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077095.913313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077095.913317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077095.913320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616675456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077095.913326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616675584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077095.913339:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077095.913345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077095/real 1769077095] req@ffff995b85ac9880 x1855007616675584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077095.913352:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616675584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077095.913360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077095.913362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077095.913364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616675584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077095.913368:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616675712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077095.913379:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077095.913385:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077095/real 1769077095] req@ffff995b85ac9c00 x1855007616675712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077095.913393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616675712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077095.913399:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077095.913401:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077095.913403:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616675712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077095.913407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616675840:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077095.913416:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077095.913422:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077095/real 1769077095] req@ffff995b85acb800 x1855007616675840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077095.913430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616675840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077095.913436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077095.913438:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077095.913439:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616675840:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077095.913473:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616675968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077095.913486:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077095.913503:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077095/real 1769077095] req@ffff995b85aca300 x1855007616675968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077095.913511:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616675968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077095.913517:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077095.913520:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077095.913522:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616675968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077095.913526:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616676096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077095.913535:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077095.913541:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077095/real 1769077095] req@ffff995b85acaa00 x1855007616676096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077095.913549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616676096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077095.913554:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077095.913556:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077095.913558:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616676096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077096.616614:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077096.616757:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077096.936612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077096.936612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077096.936613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077096.936652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077097.640536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077097.640597:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077097.960641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077097.960686:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077097.960711:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077097.960724:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077098.664545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077098.664701:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077098.984592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077098.984615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077098.984687:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077098.984709:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077099.688525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077099.688653:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077100.008581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077100.008592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077100.008599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077100.008641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077100.712560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077100.712639:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077101.032573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077101.032575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077101.032612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077101.032650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077101.032656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769077101.032659:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077101.032662:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077101.032678:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4086 00000100:00080000:3.0:1769077101.032692:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077101.032704:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077101.032752:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616676224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077101.032793:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077101.032797:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077101.032799:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077101.032803:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4086 00000100:00080000:3.0:1769077101.032808:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077101.032814:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077101.032825:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077101.032833:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616676224:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077101.032843:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616676352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077101.032874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077101.032879:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077101.032882:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077101.032889:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4086 00000100:00080000:3.0:1769077101.032896:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077101.032903:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077101.032911:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077101.032930:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077101.032931:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616676480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077101.032944:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077101.032949:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077101.032954:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077101.032959:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4086 00000100:00080000:3.0:1769077101.032965:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077101.032974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077101/real 1769077101] req@ffff995b8455b480 x1855007616676224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077101.032975:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077101.032995:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077101.033006:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616676224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077101.033018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077101.033022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077101.033026:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616676224:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077101.033047:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616676352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077101.033066:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077101.033077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077101/real 1769077101] req@ffff995b84558700 x1855007616676352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077101.033111:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616676608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077101.033123:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077101.033127:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077101.033129:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077101.033134:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4086 00000100:00080000:3.0:1769077101.033139:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077101.033159:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077101.033165:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077101.033176:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616676736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077101.033180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616676352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077101.033198:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077101.033200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077101.033206:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077101.033208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077101.033210:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077101.033215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616676352:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077101.033219:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4086 00000100:00080000:3.0:1769077101.033225:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077101.033228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616676480:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077101.033232:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077101.033238:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077101.033252:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616676864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077101.033257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077101.033266:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077101.033271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077101/real 1769077101] req@ffff995b8455a300 x1855007616676480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077101.033281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616676480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077101.033289:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077101.033291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077101.033294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616676480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077101.033304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616676608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077101.033315:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077101.033322:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077101/real 1769077101] req@ffff995b84559c00 x1855007616676608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077101.033330:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616676608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077101.033336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077101.033337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077101.033340:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616676608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077101.033343:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616676736:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077101.033354:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077101.033362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077101/real 1769077101] req@ffff995b84559880 x1855007616676736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077101.033370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616676736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077101.033377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077101.033378:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077101.033380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616676736:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077101.033384:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616676864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077101.033397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077101.033409:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769077101.033414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077101/real 1769077101] req@ffff995b8455ad80 x1855007616676864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077101.033416:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077101.033427:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4086 00000100:00100000:2.0:1769077101.033431:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616676864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077101.033435:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077101.033439:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077101.033445:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077101.033448:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616676864:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077101.033515:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077101.033526:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077101.033544:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616676992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077101.033607:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616676992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077101.033650:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077101.033665:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077101/real 1769077101] req@ffff995b84558000 x1855007616676992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077101.033676:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616676992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077101.033683:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077101.033703:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077101.033706:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616676992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077101.736593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077101.736677:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077102.056697:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077102.056703:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077102.056731:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077102.056774:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077102.760605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077102.760698:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077103.080544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077103.080592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077103.080660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077103.080678:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077103.784515:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077103.784665:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077104.104590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077104.104594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077104.104633:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077104.104636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077104.808590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077104.808707:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077105.128568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077105.128576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077105.128577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077105.128592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077105.832575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077105.832647:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077106.152556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077106.152565:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077106.152568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077106.152575:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077106.152580:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077106.152589:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4091 00000100:00080000:3.0:1769077106.152598:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077106.152601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077106.152607:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077106.152624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077106.152646:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616677120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077106.152667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077106.152670:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077106.152672:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077106.152675:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4091 00000100:00080000:3.0:1769077106.152677:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077106.152679:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077106.152688:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077106.152696:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616677248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077106.152702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077106.152704:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077106.152705:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077106.152707:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4091 00000100:00080000:3.0:1769077106.152709:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077106.152711:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077106.152713:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077106.152719:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616677376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077106.152724:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077106.152726:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077106.152727:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077106.152730:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4091 00000100:00080000:3.0:1769077106.152732:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077106.152734:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077106.152736:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077106.152743:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616677504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077106.152755:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077106.152759:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769077106.152763:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616677120:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077106.152763:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077106.152773:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4091 00000100:00080000:3.0:1769077106.152777:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077106.152782:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077106.152788:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077106.152803:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616677632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077106.152814:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077106.152817:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077106.152819:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077106.152822:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4091 00000100:00080000:3.0:1769077106.152825:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077106.152829:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077106.152832:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077106.152838:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077106.152848:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616677760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077106.152855:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077106.152858:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077106.152859:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077106.152867:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077106/real 1769077106] req@ffff995b8455bb80 x1855007616677120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077106.152867:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4091 00000100:00080000:3.0:1769077106.152875:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077106.152882:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077106.152889:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077106.152890:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616677120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077106.152904:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077106.152907:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616677888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077106.152914:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077106.152923:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616677120:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077106.152935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616677248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077106.152959:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077106.152969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077106/real 1769077106] req@ffff995b84559500 x1855007616677248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077106.152979:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616677248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077106.152985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077106.152988:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077106.152990:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616677248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077106.152995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616677376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077106.153006:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077106.153012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077106/real 1769077106] req@ffff995b84558a80 x1855007616677376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077106.153020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616677376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077106.153027:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077106.153029:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077106.153031:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616677376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077106.153051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616677504:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077106.153062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077106.153068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077106/real 1769077106] req@ffff995b8455a680 x1855007616677504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077106.153076:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616677504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077106.153083:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077106.153085:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077106.153087:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616677504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077106.153090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616677632:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077106.153098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077106.153104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077106/real 1769077106] req@ffff995b8455b100 x1855007616677632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077106.153111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616677632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077106.153117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077106.153118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077106.153120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616677632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077106.153123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616677760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077106.153132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077106.153137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077106/real 1769077106] req@ffff995b8455aa00 x1855007616677760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077106.153149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616677760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077106.153154:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077106.153156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077106.153158:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616677760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077106.153161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616677888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077106.153171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077106.153176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077106/real 1769077106] req@ffff995b84558000 x1855007616677888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077106.153183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616677888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077106.153188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077106.153190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077106.153192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616677888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077106.856595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077106.856667:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077107.176558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077107.176571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077107.176575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077107.176597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077107.880550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077107.880612:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077108.200577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077108.200577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077108.200584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077108.200630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077108.904576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077108.904652:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077109.224562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077109.224562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077109.224584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077109.224618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077109.928602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077109.928690:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077110.248556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077110.248593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077110.248614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077110.248628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077110.952605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077110.952763:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077111.272565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077111.272572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077111.272603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077111.272615:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077111.272624:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077111.272630:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077111.272636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077111.272641:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4096 00000100:00080000:0.0:1769077111.272668:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077111.272677:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077111.272736:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616678016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077111.272761:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077111.272765:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077111.272766:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077111.272770:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4096 00000100:00080000:0.0:1769077111.272775:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077111.272780:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077111.272791:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077111.272803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616678016:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077111.272812:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616678144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077111.272830:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077111.272834:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077111.272835:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077111.272839:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4096 00000100:00080000:0.0:1769077111.272843:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077111.272846:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077111.272852:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077111.272865:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616678272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077111.272872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077111.272876:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077111.272878:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077111.272880:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4096 00000100:00080000:0.0:1769077111.272883:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077111.272886:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769077111.272893:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077111.272895:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077111.272905:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616678400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077111.272915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077111.272917:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077111.272918:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077111.272922:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4096 00000100:00080000:0.0:1769077111.272925:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077111.272927:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077111.272934:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077111.272947:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077111/real 1769077111] req@ffff995b84643480 x1855007616678016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077111.272948:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616678528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077111.273025:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077111.273031:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616678016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077111.273034:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077111.273041:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077111.273048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077111.273055:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077111.273061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616678016:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077111.273064:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4096 00000100:00080000:0.0:1769077111.273073:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077111.273078:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077111.273083:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077111.273095:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616678144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077111.273097:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616678656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077111.273113:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077111.273116:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077111.273118:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077111.273124:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4096 00000800:00000100:2.0:1769077111.273125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077111.273136:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077111.273145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077111/real 1769077111] req@ffff995b84643100 x1855007616678144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077111.273147:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077111.273166:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077111.273169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616678144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077111.273180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077111.273184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077111.273189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616678144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077111.273196:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616678784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077111.273206:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616678272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077111.273240:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077111.273254:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077111/real 1769077111] req@ffff995b84642d80 x1855007616678272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077111.273263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616678272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077111.273272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077111.273274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077111.273277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616678272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077111.273282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616678400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077111.273297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077111.273303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077111/real 1769077111] req@ffff995b84641880 x1855007616678400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077111.273311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616678400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077111.273317:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077111.273319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077111.273321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616678400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077111.273325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616678528:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077111.273335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077111.273342:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077111/real 1769077111] req@ffff995b84643b80 x1855007616678528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077111.273354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616678528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077111.273361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077111.273362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077111.273364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616678528:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077111.273380:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616678656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077111.273390:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077111.273396:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077111/real 1769077111] req@ffff995b84641500 x1855007616678656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077111.273404:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616678656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077111.273409:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077111.273411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077111.273413:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616678656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077111.273417:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616678784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077111.273481:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077111.273489:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077111/real 1769077111] req@ffff995b84641180 x1855007616678784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077111.273497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616678784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077111.273508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077111.273510:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077111.273512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616678784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077111.976526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077111.976638:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077112.296552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077112.296557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077112.296591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077112.296595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077113.000579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077113.000648:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077113.320564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077113.320564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077113.320566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077113.320601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077114.024517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077114.024651:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077114.344551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077114.344565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077114.344569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077114.344608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077115.048540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077115.048604:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077115.368565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077115.368566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077115.368596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077115.368611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077116.072513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077116.072593:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077116.392541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077116.392543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077116.392548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077116.392554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077116.392609:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077116.392614:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077116.392618:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077116.392626:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4102 00000100:00080000:0.0:1769077116.392637:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077116.392643:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077116.392681:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616678912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077116.392704:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077116.392707:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077116.392708:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077116.392712:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4102 00000100:00080000:0.0:1769077116.392714:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077116.392717:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077116.392725:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077116.392735:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616679040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077116.392744:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077116.392747:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077116.392749:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077116.392752:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616678912:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077116.392754:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4102 00000100:00080000:0.0:1769077116.392759:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077116.392764:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077116.392766:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077116.392774:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616679168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077116.392780:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077116.392783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077116.392784:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077116.392787:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4102 00000100:00080000:0.0:1769077116.392789:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077116.392791:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077116.392795:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077116.392801:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616679296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077116.392807:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077116.392808:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077116.392810:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077116.392812:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4102 00000100:00080000:0.0:1769077116.392818:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077116.392820:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077116.392823:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077116.392832:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616679424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077116.392832:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077116.392841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077116.392844:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077116.392846:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077116.392849:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4102 00000100:00080000:0.0:1769077116.392852:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077116.392855:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077116.392859:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077116.392866:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616679552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077116.392871:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077116.392873:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077116.392875:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077116.392878:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4102 00000100:00080000:0.0:1769077116.392881:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077116.392885:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077116/real 1769077116] req@ffff995b84642a00 x1855007616678912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077116.392886:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077116.392890:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077116.392902:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616679680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077116.392928:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616678912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077116.392942:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077116.392948:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077116.392952:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616678912:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077116.392979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077116.393026:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077116.393041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077116/real 1769077116] req@ffff995b84643800 x1855007616679040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077116.393066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616679040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077116.393075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077116.393079:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077116.393083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077116.393090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077116.393112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077116.393125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077116/real 1769077116] req@ffff995b84640700 x1855007616679168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077116.393140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616679168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077116.393158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077116.393162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077116.393165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077116.393172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679296:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077116.393192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077116.393202:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077116/real 1769077116] req@ffff995b84642680 x1855007616679296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077116.393215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616679296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077116.393225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077116.393229:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077116.393233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077116.393250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679424:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077116.393269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077116.393279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077116/real 1769077116] req@ffff995b84641c00 x1855007616679424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077116.393291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616679424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077116.393303:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077116.393308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077116.393312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679424:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077116.393319:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616679552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077116.393385:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077116.393401:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077116/real 1769077116] req@ffff995b84640a80 x1855007616679552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077116.393414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616679552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077116.393422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077116.393426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077116.393430:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616679552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077116.393436:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616679680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077116.393477:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077116.393489:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077116/real 1769077116] req@ffff995b84640000 x1855007616679680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077116.393513:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616679680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077116.393521:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077116.393525:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077116.393528:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616679680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077117.096531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077117.096636:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077117.416556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077117.416556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077117.416558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077117.416580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077118.120564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077118.120620:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077118.440615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077118.440615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077118.440616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077118.440616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077119.144535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077119.144594:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077119.464564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077119.464564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077119.464565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077119.464569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077120.168535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077120.168586:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077120.488560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077120.488565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077120.488578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077120.488591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077121.192561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077121.192705:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077121.512601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077121.512601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077121.512608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077121.512638:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077121.512650:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077121.512656:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077121.512686:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4107 00000100:00080000:0.0:1769077121.512700:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077121.512702:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077121.512713:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077121.512759:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616679808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077121.512790:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077121.512796:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077121.512799:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077121.512806:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4107 00000100:00080000:0.0:1769077121.512811:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077121.512818:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077121.512830:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077121.512831:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616679808:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077121.512854:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616679936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077121.512870:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077121.512874:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077121.512876:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077121.512881:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4107 00000100:00080000:0.0:1769077121.512886:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077121.512890:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077121.512895:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077121.512906:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077121.512912:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616680064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077121.512930:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077121.512935:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077121.512938:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077121.512946:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4107 00000100:00080000:0.0:1769077121.512952:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077121.512957:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077121/real 1769077121] req@ffff995b84641f80 x1855007616679808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077121.512962:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077121.512979:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077121.512990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616679808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077121.513000:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616680192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077121.513002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077121.513019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077121.513037:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616679808:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077121.513038:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077121.513051:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077121.513055:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077121.513061:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4107 00000100:00080000:0.0:1769077121.513074:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077121.513079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679936:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077121.513085:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077121.513096:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077121.513107:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077121.513115:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616680320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077121.513122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077121/real 1769077121] req@ffff995b84640380 x1855007616679936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077121.513160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077121.513165:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077121.513168:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077121.513170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616679936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077121.513182:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4107 00000100:00080000:0.0:1769077121.513190:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077121.513194:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077121.513203:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077121.513207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077121.513212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077121.513221:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616680448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077121.513221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616679936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077121.513244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680064:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077121.513245:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077121.513270:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077121.513272:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077121.513278:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4107 00000100:00080000:0.0:1769077121.513284:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077121.513288:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077121.513293:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077121.513300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077121.513310:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616680576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077121.513320:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077121/real 1769077121] req@ffff995b84640e00 x1855007616680064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077121.513339:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616680064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077121.513359:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077121.513363:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077121.513367:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077121.513374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077121.513392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077121.513400:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077121/real 1769077121] req@ffff995b84640000 x1855007616680192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077121.513408:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616680192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077121.513415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077121.513417:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077121.513419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077121.513434:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680320:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077121.513445:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077121.513451:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077121/real 1769077121] req@ffff995b84640a80 x1855007616680320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077121.513482:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616680320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077121.513488:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077121.513491:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077121.513493:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680320:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077121.513498:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616680448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077121.513510:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077121.513517:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077121/real 1769077121] req@ffff995b84641c00 x1855007616680448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077121.513524:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616680448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077121.513529:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077121.513531:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077121.513533:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616680448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077121.513536:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616680576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077121.513547:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077121.513553:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077121/real 1769077121] req@ffff995b84642680 x1855007616680576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077121.513560:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616680576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077121.513565:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077121.513567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077121.513569:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616680576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077122.216614:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077122.216683:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077122.536547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077122.536547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077122.536613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077122.536614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077123.240535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077123.240632:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077123.560535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077123.560558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077123.560579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077123.560625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077124.264549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077124.264625:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077124.584554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077124.584576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077124.584578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077124.584584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077125.288630:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077125.288743:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077125.608584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077125.608610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077125.608610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077125.608648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077126.312652:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077126.312737:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077126.632556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077126.632556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077126.632557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077126.632632:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077126.632638:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769077126.632643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077126.632643:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077126.632661:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4112 00000100:00080000:2.0:1769077126.632676:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077126.632685:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077126.632728:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616680704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077126.632753:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077126.632756:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077126.632758:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077126.632761:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4112 00000100:00080000:2.0:1769077126.632764:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077126.632769:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077126.632790:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077126.632801:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616680832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077126.632810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077126.632813:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769077126.632817:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616680704:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077126.632817:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077126.632834:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4112 00000100:00080000:2.0:1769077126.632837:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077126.632839:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077126.632842:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077126.632856:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616680960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077126.632863:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077126.632866:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077126.632867:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077126.632870:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4112 00000100:00080000:2.0:1769077126.632872:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077126.632875:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077126.632879:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077126.632886:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616681088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077126.632891:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077126.632894:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077126.632896:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077126.632898:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4112 00000100:00080000:2.0:1769077126.632901:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077126.632903:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077126.632912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077126.632920:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077126.632936:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616681216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077126.632946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077126.632949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077126.632951:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077126.632955:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4112 00000100:00080000:2.0:1769077126.632959:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077126.632962:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077126.632966:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077126.632970:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077126/real 1769077126] req@ffff995b85ac8a80 x1855007616680704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077126.632980:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616681344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077126.632997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616680704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077126.632998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077126.633013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077126.633021:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077126.633038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077126.633038:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4112 00000100:00080000:2.0:1769077126.633058:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077126.633061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077126.633068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616680704:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077126.633071:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077126.633087:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077126.633101:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616681472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077126.633109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077126.633138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077126.633150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077126/real 1769077126] req@ffff995b85acb100 x1855007616680832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077126.633182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616680832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077126.633189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077126.633191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077126.633194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077126.633199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077126.633212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077126.633220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077126/real 1769077126] req@ffff995b85acaa00 x1855007616680960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077126.633228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616680960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077126.633235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077126.633238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077126.633240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616680960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077126.633244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077126.633257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077126.633263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077126/real 1769077126] req@ffff995b85aca300 x1855007616681088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077126.633271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616681088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077126.633282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077126.633284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077126.633286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077126.633290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681216:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077126.633300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077126.633306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077126/real 1769077126] req@ffff995b85acb800 x1855007616681216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077126.633313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616681216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077126.633320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077126.633322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077126.633324:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681216:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077126.633327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616681344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077126.633337:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077126.633343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077126/real 1769077126] req@ffff995b85ac9c00 x1855007616681344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077126.633350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616681344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077126.633356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077126.633358:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077126.633360:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616681344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077126.633376:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616681472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077126.633386:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077126.633393:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077126/real 1769077126] req@ffff995b85ac9880 x1855007616681472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077126.633399:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616681472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077126.633405:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077126.633407:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077126.633409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616681472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077127.336544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077127.336603:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077127.656548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077127.656548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077127.656565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077127.656601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077128.360565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077128.360617:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077128.680567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077128.680568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077128.680568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077128.680595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077129.384528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077129.384624:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077129.704587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077129.704587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077129.704628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077129.704673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077130.408569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077130.408632:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077130.728597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077130.728597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077130.728628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077130.728631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077131.432540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077131.432577:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077131.752551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077131.752551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077131.752552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077131.752585:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077131.752590:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769077131.752593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077131.752595:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077131.752605:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4117 00000100:00080000:0.0:1769077131.752616:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077131.752622:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077131.752659:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616681600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077131.752692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077131.752697:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077131.752699:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077131.752705:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4117 00000100:00080000:0.0:1769077131.752710:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077131.752716:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077131.752723:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616681600:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077131.752727:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077131.752754:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616681728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077131.752766:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077131.752769:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077131.752771:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077131.752774:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4117 00000100:00080000:0.0:1769077131.752779:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077131.752783:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077131.752788:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077131.752799:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616681856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077131.752809:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077131.752813:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077131.752816:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077131.752821:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4117 00000100:00080000:0.0:1769077131.752827:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077131.752841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681728:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077131.752842:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077131.752851:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077131.752866:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616681984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077131.752874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077131.752877:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077131.752879:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077131.752880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681856:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077131.752886:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4117 00000100:00080000:0.0:1769077131.752891:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077131.752895:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077131.752899:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077131.752910:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077131.752918:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616682112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077131.752927:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077131.752931:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077131.752932:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077131.752934:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616682112:192.168.204.132@tcp:8:kworker.0 00000100:00080000:0.0:1769077131.752939:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4117 00000100:00080000:0.0:1769077131.752944:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077131.752948:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077131.752951:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077131.752963:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616682240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077131.752977:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077131.752980:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077131.752982:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077131.752985:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4117 00000100:00080000:0.0:1769077131.752988:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077131.752992:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616682240:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077131.752994:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077131.753002:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077131.753012:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616682368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077131.753031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616682368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077132.456531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077132.456590:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077132.776561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077132.776582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077132.776584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077132.776641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077133.480524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077133.480620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077133.800533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077133.800552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077133.800559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077133.800595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077134.504523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077134.504591:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077134.824582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077134.824583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077134.824643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077134.824714:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:3.0:1769077134.824818:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:3.0:1769077134.824891:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769077134.824915:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077134.824958:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077134.824968:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077134.824977:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077134.824988:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077134.825002:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077134.825018:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769077134.825053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077131/real 1769077134] req@ffff995b84641500 x1855007616682112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077134.825101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616682112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077134.825118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077134.825125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077134.825130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616682112:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077134.825141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077131/real 1769077134] req@ffff995b84641180 x1855007616681984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077134.825156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616681984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077134.825167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077134.825172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077134.825176:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077134.825184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077131/real 1769077134] req@ffff995b84643800 x1855007616681728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077134.825199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616681728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077134.825208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077134.825213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077134.825217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077134.825226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077131/real 1769077134] req@ffff995b84642a00 x1855007616681856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077134.825240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616681856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077134.825252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077134.825257:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077134.825261:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616681856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077134.825275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077131/real 1769077134] req@ffff995b84640700 x1855007616681600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077134.825291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616681600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077134.825301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077134.825305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077134.825309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616681600:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077134.825351:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077131/real 1769077134] req@ffff995b84641880 x1855007616682368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077134.825366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616682368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077134.825383:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077134.825388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077134.825392:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616682368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077134.825401:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077131/real 1769077134] req@ffff995b84643b80 x1855007616682240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077134.825416:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616682240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077134.825429:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077134.825433:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077134.825436:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616682240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077135.528607:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077135.528676:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077135.848559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077135.848565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077135.848623:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077135.848624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077136.552559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077136.552575:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077136.872605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077136.872617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077136.872633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077136.872644:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077136.872649:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077136.872659:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4122 00000100:00080000:0.0:1769077136.872672:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077136.872679:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077136.872719:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077136.872721:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616682496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077136.872733:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077136.872754:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077136.872761:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077136.872766:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077136.872772:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4122 00000100:00080000:0.0:1769077136.872779:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077136.872785:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077136.872797:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077136.872809:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616682496:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077136.872818:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616682624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077136.872838:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077136.872842:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077136.872845:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077136.872851:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4122 00000100:00080000:0.0:1769077136.872857:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077136.872861:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077136.872868:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077136.872883:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616682752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077136.872894:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077136.872898:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077136.872901:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077136.872906:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4122 00000100:00080000:0.0:1769077136.872922:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077136.872928:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077136.872935:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077136.872938:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077136.872959:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616682880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077136.873018:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077136.873024:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769077136.873030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077136/real 1769077136] req@ffff995b84642d80 x1855007616682496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077136.873032:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077136.873049:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4122 00000100:00080000:0.0:1769077136.873061:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077136.873068:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077136.873075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616682496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077136.873081:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769077136.873104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.873119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077136.873120:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616683008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077136.873140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616682496:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077136.873150:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077136.873158:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077136.873162:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077136.873170:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4122 00000100:00080000:0.0:1769077136.873177:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077136.873186:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077136.873195:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077136.873205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616682624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077136.873217:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616683136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077136.873239:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077136.873245:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077136.873249:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077136.873253:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077136.873262:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4122 00000100:00080000:0.0:1769077136.873275:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077136.873283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077136/real 1769077136] req@ffff995b84643100 x1855007616682624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077136.873285:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077136.873313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616682624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077136.873314:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077136.873339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.873348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077136.873354:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616682624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077136.873361:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616683264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077136.873375:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616682752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077136.873418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077136.874676:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077136/real 1769077136] req@ffff995b84643480 x1855007616682752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077136.874717:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616682752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077136.874782:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.874788:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077136.874792:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616682752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077136.874802:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616682880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077136.874862:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077136.874899:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077136/real 1769077136] req@ffff995b84643b80 x1855007616682880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077136.874919:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616682880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077136.874926:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.874928:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077136.874931:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616682880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077136.874936:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683008:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077136.874953:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077136.874961:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077136/real 1769077136] req@ffff995b84641880 x1855007616683008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077136.874969:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616683008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077136.874976:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.874978:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077136.874980:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077136.875020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616683136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077136.875048:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077136.875060:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077136/real 1769077136] req@ffff995b84640700 x1855007616683136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077136.875074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616683136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077136.875102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.875105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077136.875108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616683136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077136.875113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616683264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077136.875130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077136.875138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077136/real 1769077136] req@ffff995b84642a00 x1855007616683264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077136.875147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616683264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077136.875153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.875155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077136.875157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616683264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077137.576542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077137.576600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077137.896563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077137.896602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077137.896649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077137.896660:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077138.600556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077138.600603:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077138.920605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077138.920605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077138.920618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077138.920630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077139.624562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077139.624646:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077139.944592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077139.944635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077139.944636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077139.944638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077140.648525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077140.648614:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077140.968596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077140.968596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077140.968604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077140.968668:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077141.672550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077141.672599:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077141.992537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077141.992542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077141.992562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077141.992585:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077141.992590:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077141.992594:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077141.992601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077141.992602:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4127 00000100:00080000:1.0:1769077141.992615:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077141.992622:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077141.992661:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616683392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077141.992683:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077141.992686:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077141.992687:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077141.992690:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4127 00000100:00080000:1.0:1769077141.992693:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077141.992696:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077141.992704:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077141.992711:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616683392:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077141.992716:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616683520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077141.992725:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077141.992727:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077141.992728:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077141.992730:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4127 00000100:00080000:1.0:1769077141.992733:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077141.992735:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077141.992738:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077141.992745:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616683648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077141.992751:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077141.992753:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077141.992755:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077141.992758:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4127 00000100:00080000:1.0:1769077141.992761:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077141.992763:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077141.992766:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077141.992773:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616683776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077141.992779:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077141.992781:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077141.992785:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077141.992787:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077141.992790:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4127 00000100:00080000:1.0:1769077141.992793:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077141.992796:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077141.992799:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077141.992807:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616683904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077141.992814:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077141.992816:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077141.992817:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077141.992820:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4127 00000100:00080000:1.0:1769077141.992823:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077141.992826:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077141.992829:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077141.992838:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077141/real 1769077141] req@ffff995b868c5c00 x1855007616683392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077141.992839:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616684032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077141.992861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077141.992866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077141.992869:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077141.992875:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4127 00000100:00080000:1.0:1769077141.992890:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077141.992895:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616683392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077141.992899:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077141.992914:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077141.992928:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077141.992930:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616684160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077141.992937:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077141.992947:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616683392:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077141.992977:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077141.993002:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077141.993012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077141/real 1769077141] req@ffff995b868c6a00 x1855007616683520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077141.993035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616683520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077141.993044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077141.993047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077141.993050:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077141.993057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077141.993079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077141.993091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077141/real 1769077141] req@ffff995b868c7100 x1855007616683648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077141.993104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616683648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077141.993123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077141.993126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077141.993130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077141.993137:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683776:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077141.993158:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077141.993170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077141/real 1769077141] req@ffff995b868c5500 x1855007616683776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077141.993183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616683776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077141.993193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077141.993197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077141.993200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077141.993207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683904:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077141.993227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077141.993239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077141/real 1769077141] req@ffff995b868c4700 x1855007616683904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077141.993253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616683904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077141.993263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077141.993267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077141.993271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616683904:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077141.993277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616684032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077141.993295:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077141.993303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077141/real 1769077141] req@ffff995b868c7480 x1855007616684032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077141.993311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616684032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077141.993316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077141.993318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077141.993321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616684032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077141.993324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616684160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077141.993334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077141.993340:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077141/real 1769077141] req@ffff995b868c4a80 x1855007616684160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077141.993347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616684160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077141.993352:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077141.993354:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077141.993356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616684160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077142.696544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077142.696625:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077143.016518:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077143.016530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077143.016545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077143.016570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077143.720603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077143.720683:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077144.040522:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077144.040533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077144.040546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077144.040809:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077144.744529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077144.744597:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077145.064515:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077145.064547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077145.064550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077145.064584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077145.768540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077145.768624:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077146.088521:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077146.088533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077146.088558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077146.088572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077146.792554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077146.792617:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077147.112508:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077147.112526:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077147.112527:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077147.112546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077147.112579:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077147.112587:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077147.112592:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077147.112602:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4132 00000100:00080000:3.0:1769077147.112614:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077147.112621:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077147.112659:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616684288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077147.112682:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077147.112685:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077147.112687:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077147.112690:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4132 00000100:00080000:3.0:1769077147.112694:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077147.112705:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077147.112714:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077147.112715:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616684288:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077147.112729:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616684416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077147.112743:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077147.112753:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077147.112755:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077147.112760:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4132 00000100:00080000:3.0:1769077147.112765:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077147.112770:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077147.112774:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077147.112779:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077147.112788:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616684544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077147.112796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077147.112799:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077147.112800:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077147.112803:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4132 00000100:00080000:3.0:1769077147.112807:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077147.112810:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077147.112814:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077147/real 1769077147] req@ffff995b8455ad80 x1855007616684288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077147.112828:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077147.112845:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616684672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077147.112850:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616684288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077147.112862:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077147.112869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077147.112875:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616684288:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077147.112875:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077147.112884:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077147.112886:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077147.112893:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4132 00000100:00080000:3.0:1769077147.112897:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077147.112901:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077147.112905:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077147.112913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616684416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077147.112925:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616684800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077147.112937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077147.112939:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077147.112944:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077147.112946:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077147.112951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077147/real 1769077147] req@ffff995b84559880 x1855007616684416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077147.112952:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4132 00000100:00080000:3.0:1769077147.112958:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077147.112965:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077147.112971:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077147.112973:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616684416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077147.112982:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077147.112983:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616684928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077147.112988:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077147.112996:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616684416:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077147.113000:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077147.113005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616684544:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077147.113006:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077147.113010:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077147.113015:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4132 00000100:00080000:3.0:1769077147.113018:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077147.113021:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077147.113024:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077147.113028:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077147.113036:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616685056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077147.113041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077147/real 1769077147] req@ffff995b84559c00 x1855007616684544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077147.113062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616684544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077147.113070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077147.113072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077147.113074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616684544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077147.113079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616684672:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077147.113091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077147.113097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077147/real 1769077147] req@ffff995b8455a300 x1855007616684672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077147.113105:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616684672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077147.113111:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077147.113114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077147.113117:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616684672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077147.113143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616684800:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077147.113165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077147.113178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077147/real 1769077147] req@ffff995b84558700 x1855007616684800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077147.113207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616684800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077147.113214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077147.113216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077147.113219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616684800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077147.113223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616684928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077147.113238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077147.113246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077147/real 1769077147] req@ffff995b8455b480 x1855007616684928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077147.113254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616684928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077147.113260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077147.113262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077147.113264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616684928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077147.113268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616685056:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077147.113277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077147.113283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077147/real 1769077147] req@ffff995b8455ad80 x1855007616685056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077147.113290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616685056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077147.113300:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077147.113302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077147.113304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616685056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077147.816575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077147.816633:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077148.136535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077148.136539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077148.136539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077148.136571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077148.840578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077148.840598:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077149.160577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077149.160577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077149.160585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077149.160617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077149.864540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077149.864601:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077150.184524:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077150.184526:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077150.184533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077150.184594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077150.888585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077150.888685:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077151.208534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077151.208547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077151.208578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077151.208593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077151.912548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077151.912621:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077152.232582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077152.232583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077152.232585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077152.232597:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077152.232604:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077152.232609:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077152.232618:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4137 00000100:00100000:3.0:1769077152.232620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077152.232629:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077152.232637:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077152.232687:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616685184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077152.232707:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077152.232710:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077152.232712:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077152.232717:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4137 00000100:00080000:1.0:1769077152.232721:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077152.232726:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077152.232751:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077152.232766:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616685312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077152.232784:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616685184:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077152.232798:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077152.232801:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077152.232803:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077152.232807:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4137 00000100:00080000:1.0:1769077152.232811:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077152.232815:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077152.232820:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077152.232837:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616685440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077152.232845:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077152.232847:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077152.232848:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077152.232851:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4137 00000100:00080000:1.0:1769077152.232856:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077152.232856:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077152.232862:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077152.232866:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077152.232877:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616685568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077152.232884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077152.232886:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077152.232887:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077152.232890:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4137 00000100:00080000:1.0:1769077152.232893:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077152.232894:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077152/real 1769077152] req@ffff995b868c7800 x1855007616685184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077152.232898:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077152.232905:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077152.232924:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616685696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077152.232928:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616685184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077152.232936:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077152.232944:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077152.232945:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077152.232946:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077152.232951:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4137 00000100:00080000:2.0:1769077152.232953:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077152.232956:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077152.232960:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616685184:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077152.232963:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077152.232970:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077152.232979:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616685824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077152.232987:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077152.232990:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616685312:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077152.232991:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077152.232995:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077152.233001:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4137 00000100:00080000:1.0:1769077152.233004:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077152.233007:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077152.233011:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077152.233018:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077152.233022:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616685952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077152.233033:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077152/real 1769077152] req@ffff995b868c6d80 x1855007616685312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077152.233054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616685312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077152.233060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077152.233063:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077152.233066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616685312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077152.233071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616685440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077152.233084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077152.233090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077152/real 1769077152] req@ffff995b868c6300 x1855007616685440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077152.233097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616685440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077152.233104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077152.233105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077152.233107:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616685440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077152.233111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616685568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077152.233122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077152.233127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077152/real 1769077152] req@ffff995b868c4a80 x1855007616685568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077152.233139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616685568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077152.233144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077152.233146:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077152.233148:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616685568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077152.233151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616685696:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077152.233160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077152.233166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077152/real 1769077152] req@ffff995b868c7480 x1855007616685696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077152.233173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616685696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077152.233179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077152.233180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077152.233182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616685696:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077152.233198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616685824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077152.233208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077152.233214:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077152/real 1769077152] req@ffff995b868c4700 x1855007616685824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077152.233221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616685824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077152.233231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077152.233233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077152.233234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616685824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077152.233238:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616685952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077152.233246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077152.233252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077152/real 1769077152] req@ffff995b868c5500 x1855007616685952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077152.233258:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616685952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077152.233272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077152.233274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077152.233276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616685952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077152.936525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077152.936593:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077153.256550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077153.256551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077153.256551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077153.256708:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077153.960557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077153.960622:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077154.280594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077154.280594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077154.280594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077154.280620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077154.984578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077154.984643:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077155.304544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077155.304546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077155.304547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077155.304697:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077156.008642:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077156.008650:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077156.328634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077156.328634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077156.328638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077156.328701:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077157.032533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077157.032624:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077157.352552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077157.352558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077157.352561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077157.352574:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077157.352581:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077157.352585:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077157.352586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077157.352598:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4143 00000100:00080000:1.0:1769077157.352608:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077157.352614:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077157.352666:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616686080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077157.352706:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077157.352710:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077157.352712:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077157.352717:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4143 00000100:00080000:1.0:1769077157.352723:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077157.352727:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077157.352736:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077157.352737:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616686080:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077157.352749:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616686208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077157.352759:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077157.352762:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077157.352763:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077157.352766:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4143 00000100:00080000:1.0:1769077157.352768:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077157.352781:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077157.352785:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077157.352794:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616686336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077157.352800:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077157.352803:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077157.352804:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077157.352806:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4143 00000100:00080000:1.0:1769077157.352809:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077157.352811:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769077157.352815:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769077157.352817:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077157.352828:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616686464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077157.352834:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077157.352836:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077157.352838:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077157.352841:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4143 00000100:00080000:1.0:1769077157.352843:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077157.352846:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077157.352850:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077157.352851:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077157/real 1769077157] req@ffff995b868c7100 x1855007616686080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077157.352860:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616686592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077157.352867:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077157.352871:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616686080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077157.352872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077157.352875:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077157.352884:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4143 00000100:00080000:1.0:1769077157.352888:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077157.352889:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077157.352892:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077157.352896:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769077157.352897:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077157.352902:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616686080:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077157.352909:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616686720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077157.352916:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077157.352918:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077157.352919:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077157.352922:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4143 00000100:00080000:1.0:1769077157.352924:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077157.352927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616686208:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077157.352929:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077157.352937:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077157.352947:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616686848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077157.352954:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077157.352966:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077157/real 1769077157] req@ffff995b868c6a00 x1855007616686208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077157.352975:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616686208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077157.352981:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077157.352984:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077157.352986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616686208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077157.352990:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616686336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077157.353000:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077157.353006:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077157/real 1769077157] req@ffff995b868c5c00 x1855007616686336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077157.353018:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616686336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077157.353024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077157.353026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077157.353028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616686336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077157.353032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616686464:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077157.353042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077157.353047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077157/real 1769077157] req@ffff995b868c5500 x1855007616686464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077157.353055:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616686464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077157.353061:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077157.353062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077157.353064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616686464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077157.353068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616686592:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077157.353077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077157.353083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077157/real 1769077157] req@ffff995b868c4700 x1855007616686592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077157.353090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616686592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077157.353112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077157.353114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077157.353116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616686592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077157.353119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616686720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077157.353129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077157.353134:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077157/real 1769077157] req@ffff995b868c7480 x1855007616686720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077157.353141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616686720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077157.353147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077157.353148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077157.353150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616686720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077157.353171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616686848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077157.353181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077157.353187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077157/real 1769077157] req@ffff995b868c4a80 x1855007616686848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077157.353194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616686848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077157.353199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077157.353201:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077157.353203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616686848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077158.056531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077158.056594:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077158.376595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077158.376613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077158.376685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077158.376705:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077159.080666:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077159.080734:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077159.400574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077159.400577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077159.400588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077159.400607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077160.104574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077160.104666:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077160.424589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077160.424591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077160.424611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077160.424661:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077161.128578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077161.128668:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077161.448565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077161.448565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077161.448613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077161.448662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077162.152547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077162.152630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077162.472617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077162.472621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077162.472621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077162.472631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077162.472637:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077162.472647:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077162.472648:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4148 00000100:00080000:2.0:1769077162.472668:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077162.472681:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077162.472687:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077162.472730:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616686976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077162.472774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077162.472779:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077162.472781:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077162.472786:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4148 00000100:00080000:2.0:1769077162.472792:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077162.472797:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077162.472821:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077162.472835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616686976:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077162.472839:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616687104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077162.472853:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077162.472856:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077162.472858:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077162.472862:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4148 00000100:00080000:2.0:1769077162.472867:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077162.472871:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077162.472875:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077162.472886:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616687232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077162.472892:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077162.472895:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077162.472896:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077162.472908:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4148 00000100:00080000:2.0:1769077162.472922:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077162.472925:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077162.472928:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077162.472935:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616687360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077162.472947:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077162.472949:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077162.472953:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077162.472954:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077162.472957:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4148 00000100:00080000:2.0:1769077162.472961:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077162.472964:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077162.472967:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077162.472976:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616687488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077162.472982:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077162.472984:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077162.472985:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077162.472988:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4148 00000100:00080000:2.0:1769077162.472991:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077162.472994:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077162.472997:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077162.473025:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616687616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077162.473027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077162/real 1769077162] req@ffff995b85ac8380 x1855007616686976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077162.473092:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077162.473097:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077162.473100:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077162.473103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616686976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077162.473111:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4148 00000100:00080000:2.0:1769077162.473124:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077162.473151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077162.473155:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077162.473163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:2.0:1769077162.473166:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077162.473171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616686976:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077162.473182:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616687744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077162.473205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616687104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077162.473241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077162.473255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077162/real 1769077162] req@ffff995b85acbb80 x1855007616687104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077162.473265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616687104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077162.473271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077162.473274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077162.473277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616687104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077162.473282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616687232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077162.473294:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077162.473301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077162/real 1769077162] req@ffff995b85ac9880 x1855007616687232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077162.473309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616687232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077162.473316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077162.473318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077162.473320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616687232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077162.473324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616687360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077162.473335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077162.473341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077162/real 1769077162] req@ffff995b85ac9c00 x1855007616687360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077162.473349:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616687360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077162.473355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077162.473357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077162.473362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616687360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077162.473367:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616687488:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077162.473378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077162.473384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077162/real 1769077162] req@ffff995b85acb800 x1855007616687488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077162.473396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616687488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077162.473406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077162.473409:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077162.473413:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616687488:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077162.473419:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616687616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077162.473441:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077162.473475:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077162/real 1769077162] req@ffff995b85aca300 x1855007616687616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077162.473485:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616687616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077162.473492:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077162.473494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077162.473496:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616687616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077162.473501:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616687744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077162.473523:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077162.473535:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077162/real 1769077162] req@ffff995b85acaa00 x1855007616687744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077162.473554:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616687744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077162.473580:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077162.473584:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077162.473587:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616687744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077163.176581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077163.176652:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077163.496672:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077163.496673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077163.496680:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077163.496681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769077164.200626:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077164.200676:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077164.520557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077164.520572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077164.520590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077164.520647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077165.224620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077165.224645:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077165.544550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077165.544553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077165.544629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077165.544654:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077166.248570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077166.248613:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077166.568541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077166.568560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077166.568562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077166.568614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077167.272599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077167.272601:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077167.592587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077167.592630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077167.592630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077167.592631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077167.592648:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077167.592654:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077167.592668:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4153 00000100:00080000:2.0:1769077167.592687:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077167.592688:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077167.592700:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077167.592756:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616687872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077167.592810:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077167.592817:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077167.592820:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077167.592827:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4153 00000100:00080000:2.0:1769077167.592836:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077167.592841:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077167.592852:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077167.592862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616687872:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077167.592884:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616688000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077167.592900:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077167.592904:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077167.592906:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077167.592911:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4153 00000100:00080000:2.0:1769077167.592914:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077167.592918:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077167.592923:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077167.592937:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616688128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077167.592946:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077167.592950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077167.592953:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077167.592960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077167.592962:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4153 00000100:00080000:2.0:1769077167.592970:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077167.592975:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077167.592980:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077167.592992:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616688256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077167.593001:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077167.593005:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077167.593007:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077167.593011:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4153 00000100:00080000:2.0:1769077167.593016:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077167.593020:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077167.593027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077167/real 1769077167] req@ffff995b85acb100 x1855007616687872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077167.593030:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077167.593050:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616688384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077167.593060:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077167.593061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616687872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077167.593076:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077167.593082:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077167.593099:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4153 00000100:00080000:3.0:1769077167.593102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077167.593104:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077167.593110:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077167.593112:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077167.593122:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077167.593124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616687872:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077167.593150:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616688512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077167.593167:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077167.593172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688000:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077167.593175:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077167.593183:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077167.593195:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4153 00000100:00080000:2.0:1769077167.593200:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077167.593203:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077167.593208:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077167.593222:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616688640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077167.593241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077167.593264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077167/real 1769077167] req@ffff995b85ac8a80 x1855007616688000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077167.593280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616688000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077167.593291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077167.593295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077167.593299:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077167.593305:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077167.593325:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077167.593337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077167/real 1769077167] req@ffff995b85acaa00 x1855007616688128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077167.593351:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616688128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077167.593363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077167.593368:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077167.593372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077167.593379:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077167.593397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077167.593404:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077167/real 1769077167] req@ffff995b85aca300 x1855007616688256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077167.593417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616688256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077167.593424:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077167.593426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077167.593428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077167.593432:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688384:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077167.593444:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077167.593495:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077167/real 1769077167] req@ffff995b85acb800 x1855007616688384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077167.593511:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616688384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077167.593522:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077167.593526:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077167.593530:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688384:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077167.593550:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616688512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077167.593602:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077167.593624:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077167/real 1769077167] req@ffff995b85ac9c00 x1855007616688512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077167.593647:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616688512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077167.593674:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077167.593680:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077167.593705:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616688512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077167.593717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616688640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077167.593755:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077167.593777:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077167/real 1769077167] req@ffff995b85ac9880 x1855007616688640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077167.593816:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616688640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077167.593827:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077167.593831:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077167.593835:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616688640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077168.296586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077168.296640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077168.616603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077168.616644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077168.616674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077168.616874:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077169.320606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077169.320704:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077169.640565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077169.640578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077169.640627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077169.640633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077170.344528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077170.344593:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077170.664557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077170.664585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077170.664585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077170.664592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077171.368599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077171.368664:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077171.688582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077171.688592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077171.688610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077171.688672:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077172.392527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077172.392606:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077172.712532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077172.712534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077172.712542:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077172.712548:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077172.712557:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4158 00000100:00100000:3.0:1769077172.712561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077172.712570:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077172.712579:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077172.712585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077172.712599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077172.712622:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616688768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077172.712649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077172.712654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077172.712656:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077172.712661:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4158 00000100:00080000:2.0:1769077172.712665:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077172.712670:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077172.712677:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616688768:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077172.712683:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077172.712700:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616688896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077172.712711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077172.712715:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077172.712718:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077172.712722:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4158 00000100:00080000:2.0:1769077172.712727:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077172.712732:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077172.712738:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077172.712744:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077172.712753:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616689024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077172.712766:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077172.712768:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077172.712770:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077172.712773:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4158 00000100:00080000:2.0:1769077172.712777:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077172.712780:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077172.712785:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077172.712790:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077172/real 1769077172] req@ffff995b85acbb80 x1855007616688768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077172.712798:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616689152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077172.712812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077172.712814:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077172.712816:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077172.712821:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4158 00000100:00100000:0.0:1769077172.712822:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616688768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077172.712823:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077172.712830:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077172.712836:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769077172.712841:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077172.712847:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616689280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077172.712848:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077172.712855:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616688768:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077172.712863:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077172.712866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077172.712867:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077172.712871:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4158 00000100:00080000:2.0:1769077172.712873:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077172.712875:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077172.712878:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077172.712883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077172.712889:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616689408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077172.712898:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077172.712901:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077172.712902:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077172.712905:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4158 00000100:00080000:2.0:1769077172.712908:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077172.712910:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077172.712913:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077172.712918:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077172.712925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077172/real 1769077172] req@ffff995b85ac8380 x1855007616688896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077172.712930:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616689536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077172.712945:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616688896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077172.712953:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077172.712956:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077172.712960:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616688896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077172.712965:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077172.712980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077172.712987:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077172/real 1769077172] req@ffff995b85ac9880 x1855007616689024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077172.712995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616689024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077172.713002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077172.713003:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077172.713005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077172.713009:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689152:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077172.713019:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077172.713024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077172/real 1769077172] req@ffff995b85ac9c00 x1855007616689152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077172.713032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616689152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077172.713038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077172.713039:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077172.713041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077172.713044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689280:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077172.713053:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077172.713059:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077172/real 1769077172] req@ffff995b85acb800 x1855007616689280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077172.713066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616689280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077172.713071:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077172.713073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077172.713075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077172.713086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616689408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077172.713096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077172.713102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077172/real 1769077172] req@ffff995b85aca300 x1855007616689408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077172.713109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616689408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077172.713114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077172.713116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077172.713118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616689408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077172.713121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616689536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077172.713130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077172.713136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077172/real 1769077172] req@ffff995b85acaa00 x1855007616689536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077172.713143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616689536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077172.713149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077172.713151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077172.713153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616689536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077173.416528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077173.416592:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077173.736579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077173.736589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077173.736604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077173.736612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077174.440573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077174.440629:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077174.760536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077174.760589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077174.760651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077174.760693:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077175.464544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077175.464625:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077175.784594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077175.784594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077175.784624:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077175.784632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077176.488595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077176.488679:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077176.808575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077176.808576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077176.808589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077176.808620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077177.512525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077177.512604:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077177.832545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077177.832545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077177.832556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077177.832574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077177.832607:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077177.832613:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077177.832618:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077177.832626:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4163 00000100:00080000:3.0:1769077177.832636:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077177.832641:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077177.832679:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616689664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077177.832704:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077177.832707:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077177.832709:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077177.832712:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4163 00000100:00080000:3.0:1769077177.832715:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077177.832717:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077177.832725:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077177.832735:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616689792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077177.832744:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077177.832748:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769077177.832750:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616689664:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077177.832751:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077177.832759:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4163 00000100:00080000:3.0:1769077177.832762:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077177.832764:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077177.832779:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077177.832788:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616689920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077177.832801:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077177.832804:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077177.832805:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077177.832807:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4163 00000100:00080000:3.0:1769077177.832809:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077177.832812:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077177.832816:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077177.832824:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616690048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077177.832829:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077177.832837:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077177.832841:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077177.832842:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077177.832846:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4163 00000100:00080000:3.0:1769077177.832849:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077177.832852:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077177.832855:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077177.832863:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616690176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077177.832871:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077177.832872:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077177/real 1769077177] req@ffff995b84558000 x1855007616689664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077177.832878:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077177.832882:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077177.832889:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4163 00000100:00080000:3.0:1769077177.832891:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077177.832895:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077177.832902:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077177.832912:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616689664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077177.832913:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616690304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077177.832946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077177.832950:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077177.832950:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077177.832952:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077177.832968:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077177.832972:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616689664:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077177.832976:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4163 00000100:00080000:3.0:1769077177.832980:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077177.832982:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077177.832986:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077177.832993:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616690432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077177.833006:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077177.833036:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077177.833050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077177/real 1769077177] req@ffff995b8455aa00 x1855007616689792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077177.833066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616689792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077177.833073:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077177.833075:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077177.833089:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077177.833094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077177.833108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077177.833115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077177/real 1769077177] req@ffff995b8455b100 x1855007616689920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077177.833123:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616689920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077177.833131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077177.833133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077177.833135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616689920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077177.833139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077177.833160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077177.833166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077177/real 1769077177] req@ffff995b8455a680 x1855007616690048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077177.833174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616690048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077177.833185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077177.833187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077177.833189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077177.833193:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690176:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077177.833202:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077177.833208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077177/real 1769077177] req@ffff995b84558a80 x1855007616690176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077177.833215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616690176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077177.833221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077177.833223:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077177.833225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690176:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077177.833229:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616690304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077177.833238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077177.833244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077177/real 1769077177] req@ffff995b84559500 x1855007616690304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077177.833252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616690304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077177.833257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077177.833259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077177.833262:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616690304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077177.833275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616690432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077177.833285:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077177.833291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077177/real 1769077177] req@ffff995b8455bb80 x1855007616690432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077177.833320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616690432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077177.833327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077177.833329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077177.833332:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616690432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077178.536591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077178.536680:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077178.856536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077178.856542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077178.856543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077178.856565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077179.560538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077179.560630:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077179.880607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077179.880612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077179.880614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077179.880674:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077180.584530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077180.584617:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077180.904570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077180.904570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077180.904576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077180.904603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077181.608583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077181.608650:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077181.928572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077181.928590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077181.928594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077181.928596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077182.632562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077182.632617:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077182.952525:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077182.952550:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077182.952558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077182.952559:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769077182.952565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077182.952565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077182.952566:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077182.952581:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4168 00000100:00080000:3.0:1769077182.952594:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077182.952603:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077182.952657:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616690560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077182.952683:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077182.952686:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077182.952694:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077182.952707:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4168 00000100:00080000:3.0:1769077182.952710:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077182.952714:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077182.952722:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077182.952731:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616690688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077182.952738:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077182.952740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077182.952741:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077182.952744:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4168 00000100:00080000:3.0:1769077182.952745:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077182.952747:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077182.952750:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077182.952757:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616690816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077182.952768:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077182.952771:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077182.952772:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077182.952775:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4168 00000100:00080000:3.0:1769077182.952777:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077182.952780:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077182.952784:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077182.952791:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616690944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077182.952797:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077182.952800:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077182.952802:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077182.952804:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4168 00000100:00080000:3.0:1769077182.952806:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077182.952808:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077182.952812:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077182.952819:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616691072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077182.952838:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077182.952840:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077182.952843:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077182.952849:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4168 00000100:00100000:0.0:1769077182.952851:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616690560:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077182.952852:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077182.952859:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077182.952867:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077182.952875:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616691200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077182.952885:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077182.952888:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077182.952889:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077182.952892:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4168 00000100:00080000:3.0:1769077182.952895:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077182.952897:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077182.952900:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077182.952907:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616691328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077182.952959:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077182.953038:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077182/real 1769077182] req@ffff995b8455bb80 x1855007616690560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077182.953058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616690560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077182.953067:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077182.953072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077182.953076:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616690560:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077182.953084:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077182.953138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077182.953155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077182/real 1769077182] req@ffff995b84559500 x1855007616690688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077182.953171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616690688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077182.953183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077182.953187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077182.953192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077182.953200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077182.953242:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077182.953254:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077182/real 1769077182] req@ffff995b84558a80 x1855007616690816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077182.953281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616690816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077182.953289:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077182.953291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077182.953294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077182.953299:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690944:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077182.953314:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077182.953321:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077182/real 1769077182] req@ffff995b8455a680 x1855007616690944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077182.953329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616690944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077182.953335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077182.953337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077182.953339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616690944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077182.953368:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691072:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077182.953378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077182.953385:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077182/real 1769077182] req@ffff995b8455b100 x1855007616691072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077182.953392:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616691072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077182.953399:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077182.953402:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077182.953404:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077182.953407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616691200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077182.953416:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077182.953422:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077182/real 1769077182] req@ffff995b8455aa00 x1855007616691200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077182.953435:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616691200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077182.953441:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077182.953443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077182.953444:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616691200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077182.953450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616691328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077182.953499:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077182.953513:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077182/real 1769077182] req@ffff995b84558000 x1855007616691328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077182.953526:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616691328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077182.953535:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077182.953538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077182.953542:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616691328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077183.656558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077183.656619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077183.976618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077183.976620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077183.976620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077183.976630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077184.680594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077184.680672:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077185.000580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077185.000594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077185.000619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077185.000619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077185.704562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077185.704623:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077186.024607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077186.024629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077186.024638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077186.024641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077186.728562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077186.728622:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077187.048530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077187.048537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077187.048539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077187.048540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077187.752549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077187.752619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077188.072579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077188.072604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077188.072619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077188.072620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077188.072662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077188.072668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077188.072672:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077188.072682:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4173 00000100:00080000:3.0:1769077188.072692:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077188.072697:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077188.072733:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616691456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077188.072757:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077188.072761:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077188.072762:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077188.072765:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4173 00000100:00080000:3.0:1769077188.072769:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077188.072772:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077188.072780:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077188.072789:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616691584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077188.072799:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616691456:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077188.072800:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077188.072810:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077188.072814:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077188.072821:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4173 00000100:00080000:3.0:1769077188.072823:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077188.072825:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077188.072828:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077188.072845:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616691712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077188.072852:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077188.072854:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077188.072856:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077188.072859:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4173 00000100:00080000:3.0:1769077188.072861:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077188.072864:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077188.072868:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077188.072877:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616691840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077188.072884:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077188.072887:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077188.072888:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077188.072890:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4173 00000100:00080000:3.0:1769077188.072892:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077188.072894:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077188.072899:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769077188.072905:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077188.072918:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616691968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077188.072932:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077188.072935:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077188.072938:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077188.072944:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4173 00000100:00080000:3.0:1769077188.072949:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077188.072959:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077188/real 1769077188] req@ffff995b8455ad80 x1855007616691456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077188.072960:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077188.072976:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077188.072985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616691456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077188.072993:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616692096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077188.073007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077188.073011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077188.073018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616691456:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077188.073023:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077188.073027:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077188.073030:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077188.073035:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4173 00000100:00080000:3.0:1769077188.073041:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077188.073044:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077188.073050:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077188.073056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077188.073065:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616692224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077188.073079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077188.073095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077188/real 1769077188] req@ffff995b8455b480 x1855007616691584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077188.073104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616691584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077188.073112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077188.073116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077188.073119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077188.073126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077188.073161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077188.073173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077188/real 1769077188] req@ffff995b84558700 x1855007616691712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077188.073187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616691712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077188.073200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077188.073202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077188.073205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077188.073210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691840:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077188.073226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077188.073233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077188/real 1769077188] req@ffff995b8455a300 x1855007616691840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077188.073240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616691840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077188.073247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077188.073249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077188.073251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691840:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077188.073262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691968:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077188.073274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077188.073281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077188/real 1769077188] req@ffff995b84559c00 x1855007616691968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077188.073288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616691968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077188.073294:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077188.073296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077188.073299:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616691968:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077188.073303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616692096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077188.073313:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077188.073319:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077188/real 1769077188] req@ffff995b84559880 x1855007616692096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077188.073326:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616692096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077188.073331:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077188.073333:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077188.073335:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616692096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077188.073350:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616692224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077188.073361:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077188.073367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077188/real 1769077188] req@ffff995b8455ad80 x1855007616692224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077188.073374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616692224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077188.073382:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077188.073385:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077188.073388:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616692224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077188.776560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077188.776647:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077189.096575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077189.096575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077189.096601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077189.096631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077189.800548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077189.800602:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077190.120560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077190.120560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077190.120563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077190.120613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077190.824561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077190.824638:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077191.144570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077191.144570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077191.144577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077191.144609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077191.848531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077191.848675:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077192.168536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077192.168559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077192.168566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077192.168611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077192.872594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077192.872686:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077193.192585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077193.192601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077193.192643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077193.192662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077193.192664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077193.192672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077193.192676:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077193.192689:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4178 00000100:00080000:0.0:1769077193.192702:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077193.192709:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077193.192748:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616692352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077193.192775:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077193.192778:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077193.192780:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077193.192783:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4178 00000100:00080000:0.0:1769077193.192787:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077193.192791:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077193.192801:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077193.192812:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616692480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077193.192818:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616692352:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077193.192842:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077193.192844:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077193.192846:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077193.192848:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4178 00000100:00080000:0.0:1769077193.192852:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077193.192856:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077193.192860:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077193.192870:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616692608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077193.192877:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077193.192880:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077193.192882:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077193.192885:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4178 00000100:00080000:0.0:1769077193.192887:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077193.192889:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077193.192895:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077193.192903:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616692736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077193.192911:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077193.192921:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077193.192924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077193.192926:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077193.192932:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4178 00000100:00080000:0.0:1769077193.192935:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077193.192938:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077193.192942:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077193.192956:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616692864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077193.192958:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077193/real 1769077193] req@ffff995b84643800 x1855007616692352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077193.192997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077193.193002:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077193.193005:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077193.193014:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4178 00000100:00100000:3.0:1769077193.193016:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616692352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077193.193026:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077193.193034:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077193.193056:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077193.193066:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616692992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077193.193084:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077193.193084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077193.193096:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077193.193099:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077193.193102:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077193.193110:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4178 00000100:00100000:3.0:1769077193.193115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616692352:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077193.193117:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077193.193128:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077193.193134:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077193.193145:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616693120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077193.193153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616692480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077193.193179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077193.193189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077193/real 1769077193] req@ffff995b84641180 x1855007616692480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077193.193199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616692480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077193.193205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077193.193208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077193.193210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616692480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077193.193215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616692608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077193.193227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077193.193233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077193/real 1769077193] req@ffff995b84641500 x1855007616692608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077193.193240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616692608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077193.193247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077193.193250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077193.193251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616692608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077193.193256:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616692736:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077193.193267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077193.193273:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077193/real 1769077193] req@ffff995b84642680 x1855007616692736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077193.193287:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616692736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077193.193296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077193.193300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077193.193303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616692736:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077193.193309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616692864:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077193.193330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077193.193341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077193/real 1769077193] req@ffff995b84641c00 x1855007616692864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077193.193353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616692864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077193.193360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077193.193362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077193.193364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616692864:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077193.193368:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616692992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077193.193398:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077193.193409:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077193/real 1769077193] req@ffff995b84640a80 x1855007616692992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077193.193421:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616692992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077193.193437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077193.193441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077193.193445:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616692992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077193.193497:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616693120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077193.193515:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077193.193523:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077193/real 1769077193] req@ffff995b84640000 x1855007616693120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077193.193531:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616693120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077193.193536:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077193.193539:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077193.193541:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616693120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077193.896616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077193.896708:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077194.216574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077194.216574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077194.216574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077194.216616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077194.920535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077194.920598:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077195.240557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077195.240557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077195.240569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077195.240596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077195.944598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077195.944694:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077196.264519:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077196.264523:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077196.264540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077196.264630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077196.968540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077196.968591:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077197.288553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077197.288553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077197.288558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077197.288567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077197.992612:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077197.992653:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077198.312570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077198.312571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077198.312608:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077198.312615:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077198.312622:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077198.312632:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4184 00000100:00100000:3.0:1769077198.312643:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077198.312644:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077198.312655:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077198.312679:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077198.312699:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616693248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077198.312724:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077198.312728:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077198.312729:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077198.312733:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4184 00000100:00080000:0.0:1769077198.312735:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077198.312739:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077198.312747:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077198.312755:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616693376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077198.312763:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077198.312766:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077198.312769:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077198.312770:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616693248:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077198.312774:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4184 00000100:00080000:0.0:1769077198.312778:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077198.312782:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077198.312785:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077198.312810:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616693504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077198.312817:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077198.312820:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077198.312822:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077198.312826:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4184 00000100:00080000:0.0:1769077198.312829:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077198.312832:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077198.312836:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077198.312848:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616693632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077198.312855:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077198.312857:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077198.312859:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077198.312862:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4184 00000100:00080000:0.0:1769077198.312870:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077198.312873:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077198.312876:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077198.312885:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616693760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077198.312892:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077198.312895:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077198.312897:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077198.312901:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4184 00000100:00080000:0.0:1769077198.312903:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077198.312906:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077198.312911:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077198.312923:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616693888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077198.312930:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077198.312933:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077198.312935:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077198.312940:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4184 00000100:00100000:3.0:1769077198.312942:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616693376:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077198.312943:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077198.312951:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077198.312979:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077198.312990:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616694016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077198.313052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616693504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077198.313079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616693632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077198.313101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616693760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077198.313121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616693888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077198.313149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616694016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077199.016576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077199.016689:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077199.336629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077199.336692:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077199.336692:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077199.336740:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077200.040542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077200.040644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077200.360550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077200.360550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077200.360561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077200.360597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077201.064544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077201.064598:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077201.384607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077201.384607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077201.384700:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077201.384722:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769077201.384854:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769077201.384926:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077201.384946:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077201.384998:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077201.385013:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077201.385028:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769077201.385034:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077198/real 1769077201] req@ffff995b84640380 x1855007616693376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1769077201.385049:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077201.385071:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769077201.385095:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616693376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1769077201.385096:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00080000:3.0:1769077201.385129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077201.385138:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077201.385143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616693376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077201.385155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077198/real 1769077201] req@ffff995b84641f80 x1855007616693504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077201.385189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616693504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077201.385200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077201.385206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077201.385209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616693504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077201.385218:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077198/real 1769077201] req@ffff995b84640000 x1855007616693632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077201.385233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616693632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077201.385243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077201.385248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077201.385251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616693632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077201.385260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077198/real 1769077201] req@ffff995b84640a80 x1855007616693760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077201.385275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616693760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077201.385292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077201.385296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077201.385300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616693760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077201.385309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077198/real 1769077201] req@ffff995b84641c00 x1855007616693888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077201.385322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616693888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077201.385332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077201.385336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077201.385339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616693888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077201.385347:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077198/real 1769077201] req@ffff995b84640e00 x1855007616693248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077201.385362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616693248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077201.385369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077201.385373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077201.385377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616693248:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077201.385422:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077198/real 1769077201] req@ffff995b84642680 x1855007616694016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077201.385436:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616694016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077201.385445:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077201.385450:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077201.385489:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616694016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077202.088616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077202.088683:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077202.408558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077202.408574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077202.408577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077202.408620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077203.112572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077203.112642:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077203.432550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077203.432593:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077203.432600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077203.432602:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769077203.432606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077203.432609:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077203.432622:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4189 00000100:00100000:3.0:1769077203.432634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077203.432636:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077203.432648:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077203.432692:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616694144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077203.432731:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077203.432737:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077203.432741:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077203.432748:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4189 00000100:00080000:2.0:1769077203.432754:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077203.432761:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077203.432772:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077203.432772:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616694144:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077203.432798:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616694272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077203.432813:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077203.432817:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077203.432819:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077203.432824:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4189 00000100:00080000:2.0:1769077203.432829:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077203.432834:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077203.432841:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077203.432858:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616694400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077203.432891:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077203.432901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077203.432911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077203.432915:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077203.432922:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4189 00000100:00080000:2.0:1769077203.432930:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077203.432935:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077203.432943:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077203.432959:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616694528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077203.432971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077203.432975:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077203/real 1769077203] req@ffff995b85ac8a80 x1855007616694144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077203.432978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077203.432984:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077203.433000:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4189 00000100:00080000:2.0:1769077203.433011:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077203.433022:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077203.433034:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616694144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077203.433035:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077203.433061:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616694656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077203.433061:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.433087:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077203.433088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077203.433099:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769077203.433101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616694144:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077203.433104:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077203.433118:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4189 00000100:00080000:2.0:1769077203.433125:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077203.433132:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077203.433138:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077203.433156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616694272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077203.433157:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616694784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077203.433189:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077203.433193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077203.433196:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077203.433202:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4189 00000100:00080000:2.0:1769077203.433210:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077203.433221:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769077203.433225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077203.433230:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077203.433247:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616694912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077203.433255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077203/real 1769077203] req@ffff995b85acb100 x1855007616694272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077203.433291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616694272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077203.433301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.433310:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077203.433315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616694272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077203.433325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616694400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077203.433713:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077203.433729:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077203/real 1769077203] req@ffff995b85acaa00 x1855007616694400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077203.433743:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616694400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077203.433760:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.433764:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077203.433768:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616694400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077203.433775:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616694528:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077203.433799:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077203.433811:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077203/real 1769077203] req@ffff995b85aca300 x1855007616694528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077203.433822:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616694528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077203.433832:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.433836:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077203.433840:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616694528:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077203.433847:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616694656:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077203.433867:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077203.433892:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077203/real 1769077203] req@ffff995b85acb800 x1855007616694656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077203.433905:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616694656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077203.433915:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.433919:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077203.433923:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616694656:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077203.433948:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616694784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077203.433972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077203.433985:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077203/real 1769077203] req@ffff995b85ac9c00 x1855007616694784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077203.433997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616694784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077203.434005:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.434008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077203.434012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616694784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077203.434018:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616694912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077203.434038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077203.434051:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077203/real 1769077203] req@ffff995b85ac9880 x1855007616694912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077203.434063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616694912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077203.434071:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.434075:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077203.434078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616694912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077204.136613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077204.136719:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077204.456568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077204.456589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077204.456636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077204.456712:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077205.160543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077205.160619:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077205.480596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077205.480597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077205.480601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077205.480650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077206.184587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077206.184638:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077206.504541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077206.504548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077206.504556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077206.504573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077207.208539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077207.208613:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077207.528548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077207.528550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077207.528565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077207.528585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077208.232563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077208.232621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077208.552584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077208.552608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077208.552612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077208.552649:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077208.552682:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077208.552687:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077208.552692:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077208.552701:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4194 00000100:00080000:3.0:1769077208.552715:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077208.552724:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077208.552760:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616695040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077208.552798:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077208.552804:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077208.552806:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077208.552813:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4194 00000100:00080000:3.0:1769077208.552819:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077208.552826:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077208.552827:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616695040:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769077208.552836:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077208.552859:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616695168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077208.552871:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077208.552874:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077208.552876:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077208.552879:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4194 00000100:00080000:3.0:1769077208.552886:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077208.552892:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077208.552896:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077208.552902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077208.552906:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616695296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077208.552915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077208.552918:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077208.552922:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077208.552925:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4194 00000100:00100000:2.0:1769077208.552954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077208/real 1769077208] req@ffff995b84558000 x1855007616695040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077208.552977:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616695040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077208.552985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077208.552991:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077208.552991:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077208.552997:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616695040:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077208.552999:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077208.553009:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077208.553022:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616695424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077208.553030:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616695168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077208.553056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077208.553071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077208/real 1769077208] req@ffff995b8455aa00 x1855007616695168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077208.553081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616695168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077208.553088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077208.553091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077208.553093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616695168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077208.553099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616695296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077208.553116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077208.553128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077208/real 1769077208] req@ffff995b8455b100 x1855007616695296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077208.553133:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077208.553142:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077208.553146:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077208.553153:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4194 00000100:00080000:3.0:1769077208.553158:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077208.553160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616695296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077208.553170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077208.553175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077208.553179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616695296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077208.553202:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616695424:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077208.553219:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077208.553226:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077208.553227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077208.553247:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616695552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077208.553250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077208/real 1769077208] req@ffff995b8455a680 x1855007616695424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077208.553281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616695424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077208.553285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077208.553293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077208.553294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077208.553299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077208.553299:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077208.553303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616695424:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077208.553310:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4194 00000100:00100000:2.0:1769077208.553316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616695552:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077208.553317:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077208.553326:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077208.553332:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077208.553334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077208.553344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077208/real 1769077208] req@ffff995b84558a80 x1855007616695552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077208.553349:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616695680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077208.553360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616695552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077208.553372:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077208.553373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077208.553380:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077208.553383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077208.553385:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077208.553394:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616695552:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077208.553394:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4194 00000100:00080000:3.0:1769077208.553398:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077208.553404:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077208.553409:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077208.553415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616695680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077208.553419:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616695808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077208.553450:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077208.553498:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077208/real 1769077208] req@ffff995b84559500 x1855007616695680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077208.553514:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616695680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077208.553522:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077208.553526:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077208.553529:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616695680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077208.553540:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616695808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077208.553556:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077208.553563:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077208/real 1769077208] req@ffff995b8455bb80 x1855007616695808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077208.553571:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616695808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077208.553577:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077208.553579:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077208.553581:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616695808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077209.256516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077209.256586:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077209.576554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077209.576563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077209.576566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077209.576613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077210.280549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077210.280606:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077210.600555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077210.600559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077210.600576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077210.600600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077211.304563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077211.304630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077211.624538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077211.624539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077211.624605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077211.624710:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077212.328517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077212.328586:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077212.648505:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077212.648524:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077212.648531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077212.648552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077213.352547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077213.352613:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077213.672529:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077213.672531:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077213.672535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077213.672544:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077213.672584:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077213.672590:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077213.672594:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077213.672603:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4199 00000100:00080000:3.0:1769077213.672613:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077213.672620:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077213.672656:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616695936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077213.672677:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077213.672680:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077213.672681:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077213.672684:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4199 00000100:00080000:3.0:1769077213.672687:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077213.672690:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077213.672708:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077213.672718:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616695936:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077213.672723:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616696064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077213.672736:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077213.672738:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077213.672739:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077213.672742:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4199 00000100:00080000:3.0:1769077213.672745:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077213.672748:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077213.672751:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077213.672760:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616696192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077213.672767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077213.672769:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077213.672771:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077213.672773:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4199 00000100:00080000:3.0:1769077213.672781:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077213.672784:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077213.672787:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077213.672792:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077213.672796:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616696320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077213.672804:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077213.672807:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077213.672808:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077213.672810:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4199 00000100:00080000:3.0:1769077213.672813:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077213.672815:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077213.672819:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077213.672825:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077213/real 1769077213] req@ffff995b8455bb80 x1855007616695936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077213.672829:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616696448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077213.672850:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077213.672853:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077213.672856:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077213.672857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616695936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077213.672865:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4199 00000100:00080000:3.0:1769077213.672871:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077213.672874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077213.672875:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077213.672880:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769077213.672880:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077213.672886:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616695936:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077213.672893:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616696576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077213.672900:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077213.672903:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077213.672904:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077213.672907:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4199 00000100:00080000:3.0:1769077213.672909:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077213.672911:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696064:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077213.672915:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077213.672920:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077213.672930:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616696704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077213.672946:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077213.672958:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077213/real 1769077213] req@ffff995b84559500 x1855007616696064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077213.672967:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616696064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077213.672974:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077213.672976:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077213.672979:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077213.672982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077213.672994:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077213.673000:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077213/real 1769077213] req@ffff995b84558a80 x1855007616696192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077213.673008:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616696192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077213.673015:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077213.673017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077213.673019:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077213.673022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696320:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077213.673033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077213.673038:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077213/real 1769077213] req@ffff995b8455a680 x1855007616696320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077213.673046:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616696320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077213.673051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077213.673061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077213.673065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696320:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077213.673074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696448:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077213.673096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077213.673108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077213/real 1769077213] req@ffff995b8455b100 x1855007616696448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077213.673118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616696448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077213.673125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077213.673127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077213.673129:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696448:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077213.673133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616696576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077213.673147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077213.673154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077213/real 1769077213] req@ffff995b8455aa00 x1855007616696576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077213.673161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616696576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077213.673167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077213.673169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077213.673170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616696576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077213.673182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616696704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077213.673192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077213.673199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077213/real 1769077213] req@ffff995b84558000 x1855007616696704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077213.673206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616696704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077213.673211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077213.673213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077213.673215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616696704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077214.376540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077214.376588:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077214.696525:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077214.696549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077214.696550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077214.696561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077215.400581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077215.400657:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077215.720523:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077215.720553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077215.720553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077215.720595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077216.424538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077216.424597:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077216.744560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077216.744563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077216.744573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077216.744607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077217.448559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077217.448622:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077217.768554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077217.768554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077217.768556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077217.768578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077218.472555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077218.472641:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077218.792558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077218.792558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077218.792561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077218.792562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077218.792597:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077218.792604:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077218.792609:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077218.792642:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4204 00000100:00080000:3.0:1769077218.792653:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077218.792660:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077218.792699:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616696832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077218.792723:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077218.792726:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077218.792727:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077218.792730:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4204 00000100:00080000:3.0:1769077218.792732:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077218.792735:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077218.792743:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077218.792753:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616696960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077218.792759:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616696832:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077218.792767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077218.792771:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077218.792772:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077218.792775:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4204 00000100:00080000:3.0:1769077218.792777:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077218.792779:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077218.792783:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077218.792794:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616697088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077218.792800:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077218.792803:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077218.792805:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077218.792808:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4204 00000100:00080000:3.0:1769077218.792816:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077218.792819:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077218.792823:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077218.792830:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077218.792834:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616697216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077218.792842:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077218.792845:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077218.792847:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077218.792850:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4204 00000100:00080000:3.0:1769077218.792853:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077218.792855:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077218.792857:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077218.792864:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616697344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077218.792871:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077218.792873:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077218.792875:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077218.792879:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4204 00000100:00100000:2.0:1769077218.792880:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077218/real 1769077218] req@ffff995b8455ad80 x1855007616696832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077218.792883:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077218.792890:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077218.792896:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077218.792906:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616696832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077218.792908:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616697472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077218.792932:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077218.792933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077218.792940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077218.792943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077218.792944:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077218.792950:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616696832:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077218.792951:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4204 00000100:00080000:3.0:1769077218.792960:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077218.792966:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077218.792972:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077218.792981:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077218.792988:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616697600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077218.793012:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077218.793029:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077218/real 1769077218] req@ffff995b84559880 x1855007616696960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077218.793041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616696960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077218.793048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077218.793050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077218.793053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616696960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077218.793058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077218.793073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077218.793081:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077218/real 1769077218] req@ffff995b84559c00 x1855007616697088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077218.793090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616697088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077218.793102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077218.793105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077218.793109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077218.793115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697216:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077218.793137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077218.793149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077218/real 1769077218] req@ffff995b8455a300 x1855007616697216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077218.793158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616697216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077218.793168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077218.793172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077218.793175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697216:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077218.793182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697344:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077218.793204:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077218.793216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077218/real 1769077218] req@ffff995b84558700 x1855007616697344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077218.793240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616697344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077218.793248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077218.793250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077218.793252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697344:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077218.793256:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616697472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077218.793272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077218.793280:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077218/real 1769077218] req@ffff995b8455b480 x1855007616697472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077218.793293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616697472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077218.793299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077218.793301:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077218.793304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616697472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077218.793322:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616697600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077218.793342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077218.793355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077218/real 1769077218] req@ffff995b8455ad80 x1855007616697600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077218.793368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616697600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077218.793377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077218.793381:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077218.793384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616697600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077219.496587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077219.496741:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077219.816632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077219.816650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077219.816683:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077219.816708:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077220.520540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077220.520642:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077220.840642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077220.840647:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077220.840657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077220.840660:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077221.544552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077221.544638:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077221.864619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077221.864647:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077221.864661:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077221.864691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077222.568642:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077222.568753:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077222.888666:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077222.888667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077222.888702:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077222.888720:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077223.592555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077223.592654:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077223.912590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077223.912648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077223.912653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077223.912677:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077223.912679:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077223.912688:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077223.912694:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077223.912704:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4209 00000100:00080000:1.0:1769077223.912718:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077223.912725:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077223.912768:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616697728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077223.912812:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077223.912816:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077223.912817:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077223.912826:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4209 00000100:00080000:1.0:1769077223.912829:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077223.912844:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077223.912853:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077223.912864:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616697856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077223.912889:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077223.912893:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769077223.912893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616697728:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077223.912896:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077223.912903:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4209 00000100:00080000:1.0:1769077223.912906:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077223.912909:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077223.912913:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077223.912922:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616697984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077223.912928:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077223.912931:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077223.912932:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077223.912935:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4209 00000100:00080000:1.0:1769077223.912939:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077223.912941:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077223.912945:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077223.912953:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616698112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077223.912959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077223.912961:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077223.912964:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077223.912967:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4209 00000100:00080000:1.0:1769077223.912969:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077223.912972:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077223.912976:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077223.912979:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077223.912987:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616698240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077223.912996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077223.912999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077223.913001:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077223.913004:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4209 00000100:00080000:1.0:1769077223.913007:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077223.913010:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077223.913015:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077223.913024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077223/real 1769077223] req@ffff995b868c6300 x1855007616697728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077223.913027:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616698368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077223.913067:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077223.913070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077223.913071:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077223.913076:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616697728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077223.913078:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4209 00000100:00080000:1.0:1769077223.913084:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077223.913090:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077223.913092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769077223.913094:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077223.913099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077223.913104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616697728:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077223.913107:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616698496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077223.913129:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077223.913151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077223.913159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077223/real 1769077223] req@ffff995b868c6d80 x1855007616697856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077223.913168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616697856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077223.913174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077223.913176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077223.913179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077223.913182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077223.913192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077223.913199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077223/real 1769077223] req@ffff995b868c7800 x1855007616697984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077223.913207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616697984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077223.913215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077223.913216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077223.913218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616697984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077223.913222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616698112:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077223.913233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077223.913238:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077223/real 1769077223] req@ffff995b868c4a80 x1855007616698112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077223.913246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616698112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077223.913252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077223.913253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077223.913255:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616698112:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077223.913259:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616698240:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077223.913268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077223.913281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077223/real 1769077223] req@ffff995b868c7480 x1855007616698240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077223.913288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616698240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077223.913296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077223.913298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077223.913300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616698240:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077223.913303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616698368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077223.913312:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077223.913318:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077223/real 1769077223] req@ffff995b868c4700 x1855007616698368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077223.913334:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616698368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077223.913352:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077223.913354:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077223.913356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616698368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077223.913360:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616698496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077223.913371:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077223.913378:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077223/real 1769077223] req@ffff995b868c5500 x1855007616698496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077223.913387:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616698496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077223.913392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077223.913394:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077223.913397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616698496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077224.616618:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077224.616713:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077224.936576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077224.936585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077224.936664:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077224.936688:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077225.640566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077225.640653:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077225.960574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077225.960614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077225.960617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077225.960674:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077226.664533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077226.664614:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077226.984566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077226.984572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077226.984684:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077226.984706:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077227.688554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077227.688647:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077228.008577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077228.008578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077228.008594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077228.008595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077228.712529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077228.712663:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077229.032599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077229.032618:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077229.032627:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077229.032633:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077229.032647:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4214 00000100:00100000:2.0:1769077229.032652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077229.032661:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077229.032665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077229.032675:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077229.032719:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616698624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077229.032728:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077229.032750:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077229.032755:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077229.032757:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077229.032761:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4214 00000100:00080000:3.0:1769077229.032768:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077229.032772:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077229.032781:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077229.032791:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616698624:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077229.032794:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616698752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077229.032805:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077229.032807:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077229.032808:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077229.032811:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4214 00000100:00080000:3.0:1769077229.032815:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077229.032817:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077229.032821:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077229.032829:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616698880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077229.032846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077229.032849:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077229.032850:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077229.032855:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4214 00000100:00080000:3.0:1769077229.032858:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077229.032862:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077229.032867:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077229.032883:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616699008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077229.032884:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077229.032897:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077229.032900:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077229.032902:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077229.032907:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4214 00000100:00080000:3.0:1769077229.032910:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077229.032913:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077229.032917:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077229.032929:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616699136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077229.032933:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077229/real 1769077229] req@ffff995b84558000 x1855007616698624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077229.032945:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077229.032950:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077229.032951:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077229.032955:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4214 00000100:00080000:3.0:1769077229.032958:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077229.032961:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616698624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077229.032964:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077229.032971:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077229.032976:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077229.032981:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616699264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077229.032983:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077229.032991:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616698624:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077229.033002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077229.033004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077229.033006:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077229.033009:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4214 00000100:00080000:3.0:1769077229.033012:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077229.033014:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077229.033016:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077229.033024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616698752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077229.033026:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616699392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077229.033049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077229.033065:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077229/real 1769077229] req@ffff995b8455aa00 x1855007616698752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077229.033079:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616698752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077229.033088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077229.033092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077229.033118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616698752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077229.033125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616698880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077229.033144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077229.033152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077229/real 1769077229] req@ffff995b8455b100 x1855007616698880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077229.033160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616698880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077229.033168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077229.033170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077229.033173:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616698880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077229.033177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699008:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077229.033190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077229.033196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077229/real 1769077229] req@ffff995b8455a680 x1855007616699008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077229.033204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616699008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077229.033211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077229.033213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077229.033214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077229.033218:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699136:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077229.033228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077229.033234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077229/real 1769077229] req@ffff995b84558a80 x1855007616699136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077229.033241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616699136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077229.033247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077229.033249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077229.033251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699136:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077229.033255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616699264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077229.033265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077229.033279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077229/real 1769077229] req@ffff995b84559500 x1855007616699264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077229.033286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616699264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077229.033293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077229.033295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077229.033297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616699264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077229.033312:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616699392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077229.033322:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077229.033329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077229/real 1769077229] req@ffff995b8455bb80 x1855007616699392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077229.033336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616699392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077229.033342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077229.033344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077229.033345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616699392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077229.736532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077229.736619:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077230.056549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077230.056571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077230.056603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077230.056636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077230.760519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077230.760666:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077231.080553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077231.080559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077231.080563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077231.080576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077231.784577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077231.784634:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077232.104601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077232.104601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077232.104619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077232.104619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077232.808598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077232.808631:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077233.128561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077233.128562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077233.128576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077233.128677:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077233.832543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077233.832606:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077234.152587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077234.152588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077234.152588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077234.152588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077234.152599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077234.152604:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077234.152613:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4219 00000100:00080000:2.0:1769077234.152625:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077234.152632:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077234.152636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077234.152681:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616699520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077234.152743:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077234.152747:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077234.152749:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077234.152753:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4219 00000100:00080000:2.0:1769077234.152757:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077234.152760:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077234.152770:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077234.152780:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616699520:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077234.152785:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616699648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077234.152796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077234.152800:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077234.152801:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077234.152804:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4219 00000100:00080000:2.0:1769077234.152807:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077234.152810:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077234.152813:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077234.152826:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616699776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077234.152833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077234.152836:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077234.152839:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077234.152841:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4219 00000100:00080000:2.0:1769077234.152845:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077234.152848:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077234.152855:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077234.152856:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077234.152867:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616699904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077234.152875:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077234.152878:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077234.152880:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077234.152883:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4219 00000100:00080000:2.0:1769077234.152885:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077234.152887:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077234.152891:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077234.152900:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616700032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077234.152908:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077234/real 1769077234] req@ffff995b85ac8380 x1855007616699520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077234.152910:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077234.152918:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077234.152922:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077234.152932:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4219 00000100:00080000:2.0:1769077234.152937:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077234.152941:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616699520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077234.152947:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769077234.152955:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769077234.152957:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077234.152961:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077234.152968:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616699520:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077234.152979:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616700160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077234.152998:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699648:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077234.153006:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077234.153011:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077234.153013:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077234.153017:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4219 00000100:00080000:2.0:1769077234.153021:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077234.153025:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077234.153039:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077234.153044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077234.153055:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616700288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077234.153064:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077234/real 1769077234] req@ffff995b85acbb80 x1855007616699648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077234.153098:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616699648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077234.153109:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077234.153113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077234.153118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077234.153126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077234.153152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077234.153184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077234/real 1769077234] req@ffff995b85ac9880 x1855007616699776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077234.153204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616699776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077234.153229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077234.153235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077234.153241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077234.153268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699904:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077234.153295:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077234.153309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077234/real 1769077234] req@ffff995b85ac9c00 x1855007616699904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077234.153322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616699904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077234.153332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077234.153336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077234.153340:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616699904:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077234.153391:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700032:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077234.153410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077234.153419:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077234/real 1769077234] req@ffff995b85acb800 x1855007616700032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077234.153430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616700032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077234.153438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077234.153440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077234.153444:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700032:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077234.153487:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616700160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077234.153519:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077234.153528:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077234/real 1769077234] req@ffff995b85aca300 x1855007616700160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077234.153537:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616700160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077234.153544:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077234.153546:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077234.153549:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616700160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077234.153553:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616700288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077234.153564:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077234.153571:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077234/real 1769077234] req@ffff995b85acaa00 x1855007616700288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077234.153580:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616700288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077234.153585:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077234.153587:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077234.153589:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616700288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077234.856546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077234.856585:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077235.176537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077235.176550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077235.176572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077235.176595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077235.880547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077235.880658:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077236.200574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077236.200574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077236.200575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077236.200723:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077236.904544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077236.904616:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077237.224541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077237.224548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077237.224586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077237.224589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077237.928529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077237.928569:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077238.248588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077238.248588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077238.248589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077238.248623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077238.952601:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077238.952650:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077239.272550:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077239.272551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077239.272551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077239.272559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077239.272561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077239.272568:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077239.272576:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4224 00000100:00080000:2.0:1769077239.272587:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077239.272594:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077239.272599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077239.272632:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616700416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077239.272654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077239.272658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077239.272659:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077239.272662:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4224 00000100:00080000:2.0:1769077239.272665:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077239.272668:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077239.272677:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077239.272687:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616700416:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077239.272688:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616700544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077239.272701:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077239.272703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077239.272704:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077239.272707:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4224 00000100:00080000:2.0:1769077239.272710:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077239.272712:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077239.272716:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077239.272726:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616700672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077239.272733:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077239.272736:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077239.272737:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077239.272739:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4224 00000100:00080000:2.0:1769077239.272741:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077239.272744:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077239.272748:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077239.272754:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616700800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077239.272760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077239.272763:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077239.272765:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077239.272768:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077239.272771:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4224 00000100:00080000:2.0:1769077239.272775:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077239.272777:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077239.272781:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077239.272790:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616700928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077239.272796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077239.272799:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077239.272801:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077239.272804:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077239/real 1769077239] req@ffff995b85acb100 x1855007616700416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077239.272807:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4224 00000100:00080000:2.0:1769077239.272813:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077239.272819:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077239.272826:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077239.272831:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616700416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077239.272837:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616701056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077239.272846:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077239.272848:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077239.272851:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077239.272853:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077239.272858:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077239.272859:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4224 00000100:00080000:2.0:1769077239.272866:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077239.272868:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616700416:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077239.272872:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077239.272878:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077239.272885:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616701184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077239.272902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077239.272931:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077239.272946:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077239/real 1769077239] req@ffff995b85ac8a80 x1855007616700544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077239.272956:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616700544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077239.272962:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077239.272964:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077239.272966:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077239.272973:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077239.272987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077239.272994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077239/real 1769077239] req@ffff995b85acaa00 x1855007616700672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077239.273002:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616700672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077239.273009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077239.273010:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077239.273012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077239.273016:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700800:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077239.273029:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077239.273040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077239/real 1769077239] req@ffff995b85aca300 x1855007616700800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077239.273063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616700800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077239.273073:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077239.273077:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077239.273080:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077239.273087:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700928:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077239.273104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077239.273112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077239/real 1769077239] req@ffff995b85acb800 x1855007616700928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077239.273120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616700928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077239.273126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077239.273128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077239.273131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616700928:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077239.273134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616701056:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077239.273144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077239.273159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077239/real 1769077239] req@ffff995b85ac9c00 x1855007616701056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077239.273175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616701056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077239.273180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077239.273182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077239.273184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616701056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077239.273187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616701184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077239.273197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077239.273203:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077239/real 1769077239] req@ffff995b85ac9880 x1855007616701184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077239.273216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616701184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077239.273227:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077239.273229:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077239.273231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616701184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077239.976527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077239.976649:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077240.296560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077240.296565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077240.296595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077240.296596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077241.000570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077241.000642:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077241.320535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077241.320556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077241.320568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077241.320575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077242.024535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077242.024601:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077242.344565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077242.344566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077242.344566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077242.344607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077243.048561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077243.048677:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077243.368531:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077243.368533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077243.368534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077243.368550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077244.072518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077244.072584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077244.392620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077244.392622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077244.392628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077244.392653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077244.392660:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077244.392666:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077244.392674:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4230 00000100:00080000:2.0:1769077244.392687:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077244.392695:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077244.392710:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077244.392737:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616701312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077244.392762:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077244.392765:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077244.392767:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077244.392770:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4230 00000100:00080000:2.0:1769077244.392773:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077244.392777:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077244.392789:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077244.392807:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616701440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077244.392814:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616701312:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077244.392841:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077244.392846:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077244.392848:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077244.392855:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4230 00000100:00080000:2.0:1769077244.392860:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077244.392865:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077244.392872:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077244.392890:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616701568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077244.392902:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077244.392905:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077244.392911:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077244.392915:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077244.392938:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4230 00000100:00080000:2.0:1769077244.392943:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077244.392951:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077244.392959:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077244.392967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077244/real 1769077244] req@ffff995b85acbb80 x1855007616701312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077244.392980:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616701696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077244.392987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616701312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077244.393025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077244.393028:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077244.393034:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077244.393036:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077244.393039:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077244.393042:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616701312:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077244.393051:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4230 00000100:00080000:2.0:1769077244.393057:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077244.393062:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077244.393067:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077244.393075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616701440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077244.393083:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616701824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077244.393097:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077244.393099:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077244.393101:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769077244.393109:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077244.393118:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4230 00000100:00080000:2.0:1769077244.393124:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077244.393131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077244/real 1769077244] req@ffff995b85ac8380 x1855007616701440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077244.393132:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077244.393142:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077244.393164:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616701952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077244.393166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616701440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077244.393209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077244.393219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077244.393226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616701440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077244.393237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616701568:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077244.393239:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077244.393245:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077244.393248:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077244.393253:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4230 00000800:00000100:0.0:1769077244.393271:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077244.393278:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077244.393285:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077244.393289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077244/real 1769077244] req@ffff995b85ac9880 x1855007616701568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077244.393298:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077244.393304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616701568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077244.393314:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616702080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077244.393349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077244.393356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077244.393361:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616701568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077244.393370:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616701696:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077244.393387:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077244.393397:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077244/real 1769077244] req@ffff995b85ac9c00 x1855007616701696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077244.393406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616701696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077244.393418:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077244.393420:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077244.393422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616701696:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077244.393438:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616701824:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077244.393451:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077244.393493:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077244/real 1769077244] req@ffff995b85acb800 x1855007616701824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077244.393503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616701824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077244.393511:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077244.393515:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077244.393518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616701824:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077244.393523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616701952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077244.393540:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077244.393552:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077244/real 1769077244] req@ffff995b85aca300 x1855007616701952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077244.393560:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616701952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077244.393566:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077244.393569:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077244.393571:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616701952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077244.393577:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616702080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077244.393594:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077244.393614:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077244/real 1769077244] req@ffff995b85acaa00 x1855007616702080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077244.393622:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616702080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077244.393628:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077244.393632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077244.393639:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616702080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077245.096546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077245.096615:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077245.416529:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077245.416530:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077245.416541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077245.416548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077246.120529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077246.120640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077246.440580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077246.440580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077246.440611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077246.440638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077247.144553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077247.144621:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077247.464545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077247.464545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077247.464552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077247.464635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077248.168524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077248.168566:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077248.488545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077248.488585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077248.488586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077248.488610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077249.192562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077249.192608:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077249.512594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077249.512594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077249.512630:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077249.512644:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077249.512650:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077249.512664:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4235 00000100:00080000:2.0:1769077249.512678:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077249.512688:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077249.512719:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077249.512733:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077249.512736:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616702208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077249.512772:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077249.512777:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077249.512780:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077249.512786:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4235 00000100:00080000:2.0:1769077249.512793:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077249.512798:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077249.512808:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616702208:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077249.512810:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077249.512836:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616702336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077249.512851:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077249.512854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077249.512856:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077249.512860:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4235 00000100:00080000:2.0:1769077249.512865:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077249.512869:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077249.512878:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077249.512892:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616702464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077249.512902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077249.512906:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1769077249.512912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077249.512913:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077249.512924:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4235 00000100:00080000:2.0:1769077249.512930:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077249.512935:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077249.512941:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077249.512955:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616702592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077249.512966:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077249.512970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077249.512972:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077249.512979:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4235 00000100:00100000:1.0:1769077249.512980:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077249/real 1769077249] req@ffff995b85ac8a80 x1855007616702208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077249.512987:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077249.513003:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077249.513011:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077249.513012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616702208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077249.513031:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077249.513038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077249.513038:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616702720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077249.513047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616702208:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077249.513069:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077249.513073:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077249.513076:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077249.513082:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4235 00000100:00080000:2.0:1769077249.513086:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077249.513090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616702336:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077249.513125:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077249.513133:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077249.513148:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077249.513154:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616702848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077249.513168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077249/real 1769077249] req@ffff995b85acb100 x1855007616702336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077249.513169:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077249.513187:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077249.513194:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077249.513204:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4235 00000100:00100000:1.0:1769077249.513208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616702336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077249.513212:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077249.513228:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769077249.513231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769077249.513239:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769077249.513240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077249.513249:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616702336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077249.513264:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616702464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077249.513265:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616702976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077249.513302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077249.513319:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077249/real 1769077249] req@ffff995b85acaa00 x1855007616702464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077249.513333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616702464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077249.513345:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077249.513349:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077249.513353:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616702464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077249.513360:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616702592:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077249.513382:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077249.513402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077249/real 1769077249] req@ffff995b85aca300 x1855007616702592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077249.513417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616702592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077249.513433:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077249.513436:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077249.513440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616702592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077249.513447:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616702720:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077249.513484:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077249.513493:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077249/real 1769077249] req@ffff995b85acb800 x1855007616702720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077249.513501:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616702720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077249.513508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077249.513510:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077249.513512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616702720:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077249.513528:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616702848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077249.513540:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077249.513547:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077249/real 1769077249] req@ffff995b85ac9c00 x1855007616702848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077249.513554:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616702848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077249.513559:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077249.513561:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077249.513564:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616702848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077249.513568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616702976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077249.513578:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077249.513584:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077249/real 1769077249] req@ffff995b85ac9880 x1855007616702976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077249.513591:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616702976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077249.513597:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077249.513598:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077249.513600:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616702976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077250.216525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077250.216579:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077250.536549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077250.536549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077250.536935:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077250.536936:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077251.240541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077251.240606:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077251.560557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077251.560557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077251.560706:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077251.560706:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077252.264535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077252.264602:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077252.584524:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077252.584528:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077252.584556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077252.584571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077253.288510:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077253.288589:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077253.608591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077253.608601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077253.608621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077253.608629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077254.312691:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077254.312731:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077254.632530:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077254.632541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077254.632545:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077254.632546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077254.632552:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077254.632558:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077254.632568:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4240 00000100:00100000:0.0:1769077254.632578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077254.632581:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077254.632591:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077254.632633:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616703104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077254.632655:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077254.632659:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077254.632668:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077254.632673:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4240 00000100:00080000:2.0:1769077254.632678:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077254.632683:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077254.632689:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616703104:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077254.632692:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077254.632705:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616703232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077254.632715:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077254.632719:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077254.632721:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077254.632725:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4240 00000100:00080000:2.0:1769077254.632728:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077254.632732:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077254.632737:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077254.632747:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616703360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077254.632755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077254.632759:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077254.632760:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077254.632762:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077254.632764:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4240 00000100:00080000:2.0:1769077254.632767:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077254.632771:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077254.632775:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077254.632785:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616703488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077254.632793:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077254.632797:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077254.632798:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077254.632801:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4240 00000100:00100000:1.0:1769077254.632802:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077254/real 1769077254] req@ffff995b85ac8380 x1855007616703104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077254.632804:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077254.632808:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077254.632813:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077254.632820:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616703104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077254.632821:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616703616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077254.632831:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077254.632835:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077254.632836:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077254.632839:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077254.632840:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4240 00000100:00080000:2.0:1769077254.632843:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077254.632844:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077254.632846:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077254.632848:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616703104:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077254.632851:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077254.632859:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616703744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077254.632866:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077254.632870:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769077254.632871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616703232:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077254.632872:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077254.632875:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4240 00000100:00080000:2.0:1769077254.632878:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077254.632882:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077254.632887:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077254.632891:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077254.632896:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616703872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077254.632903:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077254/real 1769077254] req@ffff995b85acbb80 x1855007616703232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077254.632915:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616703232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077254.632924:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077254.632928:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077254.632931:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616703232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077254.632938:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616703360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077254.632957:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077254.632965:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077254/real 1769077254] req@ffff995b85ac9880 x1855007616703360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077254.632973:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616703360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077254.632980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077254.632982:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077254.632985:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616703360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077254.632997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616703488:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077254.633032:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077254.633040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077254/real 1769077254] req@ffff995b85ac9c00 x1855007616703488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077254.633048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616703488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077254.633054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077254.633056:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077254.633058:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616703488:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077254.633062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616703616:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077254.633073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077254.633079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077254/real 1769077254] req@ffff995b85acb800 x1855007616703616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077254.633086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616703616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077254.633092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077254.633094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077254.633096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616703616:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077254.633099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616703744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077254.633109:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077254.633115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077254/real 1769077254] req@ffff995b85aca300 x1855007616703744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077254.633122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616703744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077254.633128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077254.633129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077254.633131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616703744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077254.633144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616703872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077254.633153:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077254.633159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077254/real 1769077254] req@ffff995b85acaa00 x1855007616703872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077254.633165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616703872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077254.633170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077254.633172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077254.633174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616703872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077255.336558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077255.336624:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077255.656606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077255.656623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077255.656628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077255.656636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077256.360542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077256.360605:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077256.680523:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077256.680523:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077256.680536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077256.680629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077257.384529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077257.384633:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077257.704539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077257.704542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077257.704561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077257.704572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077258.408559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077258.408617:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077258.728565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077258.728602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077258.728603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077258.728624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077259.432543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077259.432605:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077259.752577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077259.752577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077259.752581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077259.752613:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077259.752619:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769077259.752620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077259.752623:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077259.752633:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4245 00000100:00080000:0.0:1769077259.752644:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077259.752651:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077259.752704:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616704000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077259.752727:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077259.752730:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077259.752731:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077259.752734:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4245 00000100:00080000:0.0:1769077259.752738:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077259.752741:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077259.752750:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077259.752758:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616704000:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077259.752763:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616704128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077259.752773:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077259.752776:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077259.752778:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077259.752781:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4245 00000100:00080000:0.0:1769077259.752784:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077259.752786:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077259.752789:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077259.752799:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616704256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077259.752805:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077259.752808:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077259.752809:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077259.752812:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4245 00000100:00080000:0.0:1769077259.752829:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077259.752836:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769077259.752836:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077259.752843:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077259.752852:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616704384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077259.752859:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077259.752862:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077259.752863:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077259.752866:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4245 00000100:00080000:0.0:1769077259.752868:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077259.752871:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077259.752876:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077259.752882:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077259/real 1769077259] req@ffff995b84641500 x1855007616704000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077259.752887:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616704512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077259.752919:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077259.752938:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616704000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077259.752941:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077259.752946:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077259.752955:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4245 00000100:00080000:1.0:1769077259.752956:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077259.752963:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077259.752964:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077259.752971:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616704000:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077259.752975:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077259.752984:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077259.752995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616704128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077259.753002:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616704640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077259.753016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077259.753021:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077259.753022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077259.753025:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077259.753031:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4245 00000100:00080000:0.0:1769077259.753035:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077259.753036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077259/real 1769077259] req@ffff995b84641180 x1855007616704128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077259.753041:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077259.753051:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077259.753056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616704128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077259.753065:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616704768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077259.753065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077259.753073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077259.753080:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616704128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077259.753091:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616704256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077259.753105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077259.753112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077259/real 1769077259] req@ffff995b84643800 x1855007616704256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077259.753124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616704256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077259.753131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077259.753133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077259.753135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616704256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077259.753138:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616704384:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077259.753149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077259.753154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077259/real 1769077259] req@ffff995b84642a00 x1855007616704384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077259.753162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616704384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077259.753168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077259.753170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077259.753172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616704384:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077259.753175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616704512:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077259.753184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077259.753190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077259/real 1769077259] req@ffff995b84640700 x1855007616704512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077259.753197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616704512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077259.753207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077259.753210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077259.753212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616704512:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077259.753224:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616704640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077259.753233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077259.753239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077259/real 1769077259] req@ffff995b84641880 x1855007616704640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077259.753246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616704640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077259.753252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077259.753254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077259.753256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616704640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077259.753259:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616704768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077259.753268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077259.753274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077259/real 1769077259] req@ffff995b84643b80 x1855007616704768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077259.753280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616704768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077259.753285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077259.753287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077259.753289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616704768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077260.456564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077260.456645:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077260.776549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077260.776553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077260.776557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077260.776566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077261.480526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077261.480582:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077261.800572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077261.800572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077261.800613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077261.800668:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077262.504600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077262.504696:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077262.824554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077262.824600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077262.824604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077262.824628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077263.528616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077263.528690:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077263.848542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077263.848576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077263.848611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077263.848615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077264.552575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077264.552722:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077264.872596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077264.872598:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077264.872602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077264.872602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077264.872615:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077264.872620:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077264.872632:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4250 00000100:00080000:0.0:1769077264.872646:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077264.872654:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077264.872702:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616704896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077264.872712:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077264.872740:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077264.872745:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077264.872747:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077264.872752:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4250 00000100:00080000:0.0:1769077264.872756:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077264.872760:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077264.872770:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077264.872791:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616705024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077264.872791:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616704896:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077264.872823:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077264.872828:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077264.872831:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077264.872867:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4250 00000100:00080000:0.0:1769077264.872900:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077264.872906:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077264.872912:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077264.872934:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616705152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077264.872950:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077264.872954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077264.872956:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077264.872961:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4250 00000100:00080000:0.0:1769077264.872965:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077264.872969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705024:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077264.872976:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077264.873000:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077264.873020:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616705280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077264.873033:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077264.873037:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077264.873040:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077264.873046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705152:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077264.873047:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4250 00000100:00080000:0.0:1769077264.873058:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077264.873063:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077264.873070:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077264.873089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077264.873098:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616705408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077264.873134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077264.873139:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077264.873143:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077264.873154:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4250 00000100:00080000:0.0:1769077264.873166:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077264.873173:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705408:192.168.204.132@tcp:8:kworker.0 00000100:00080000:0.0:1769077264.873175:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077264.873190:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077264.873210:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616705536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077264.873228:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077264.873231:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077264.873233:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077264.873253:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4250 00000100:00080000:0.0:1769077264.873257:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077264.873271:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077264.873275:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077264.873285:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616705664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077264.873295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616705536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077264.873341:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616705664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077265.576545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077265.576644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077265.896614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077265.896670:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077265.896777:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077265.896790:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077266.600573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077266.600657:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077266.920636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077266.920641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077266.920650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077266.920693:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077267.624594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077267.624722:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077267.944617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077267.944701:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077267.944702:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077267.944896:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000400:00000100:3.0:1769077267.945015:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:3.0:1769077267.945070:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769077267.945093:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077267.945160:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077267.945174:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077267.945188:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077267.945203:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077267.945217:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077267.945228:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:1.0:1769077267.945341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077264/real 1769077267] req@ffff995b84642a00 x1855007616705664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077267.945379:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616705664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077267.945401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077267.945410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077267.945417:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616705664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077267.945430:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077264/real 1769077267] req@ffff995b84640700 x1855007616705536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077267.945604:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616705536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077267.945622:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077267.945630:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077267.945636:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616705536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077267.945649:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077264/real 1769077267] req@ffff995b84641880 x1855007616705408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077267.945672:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616705408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077267.945689:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077267.945695:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077267.945700:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077267.945712:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077264/real 1769077267] req@ffff995b84643b80 x1855007616705280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077267.945732:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616705280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077267.945746:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077267.945752:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077267.945758:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077267.945817:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077264/real 1769077267] req@ffff995b84643100 x1855007616705024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077267.945848:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616705024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077267.945862:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077267.945869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077267.945875:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077267.945885:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077264/real 1769077267] req@ffff995b84642d80 x1855007616705152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077267.945903:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616705152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077267.945919:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077267.945925:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077267.945931:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077267.945941:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077264/real 1769077267] req@ffff995b84643480 x1855007616704896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077267.945962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616704896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077267.945974:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077267.945980:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077267.945986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616704896:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077268.648638:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077268.648757:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077268.968625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077268.968626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077268.968689:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077268.968877:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077269.672729:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077269.672793:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077269.992589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077269.992595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077269.992620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077269.992692:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077269.992701:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769077269.992703:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077269.992707:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077269.992720:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4255 00000100:00080000:1.0:1769077269.992734:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077269.992745:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077269.992832:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616705792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077269.992890:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077269.992895:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077269.992898:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077269.992904:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4255 00000100:00080000:1.0:1769077269.992911:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077269.992918:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077269.992952:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077269.992965:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616705792:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077269.992973:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616705920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077269.992989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077269.992995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077269.992998:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077269.993005:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4255 00000100:00080000:1.0:1769077269.993011:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077269.993018:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077269.993027:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077269.993047:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616706048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077269.993066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077269.993072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077269.993076:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077269.993082:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077269.993091:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4255 00000100:00080000:1.0:1769077269.993110:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077269.993126:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077269.993136:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077269.993156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077269/real 1769077269] req@ffff995b868c5c00 x1855007616705792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077269.993163:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616706176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077269.993191:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077269.993205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077269.993210:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077269.993217:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4255 00000100:00100000:0.0:1769077269.993234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616705792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077269.993234:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077269.993249:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077269.993265:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769077269.993268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077269.993282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077269.993291:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616706304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077269.993294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616705792:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077269.993320:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077269.993326:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077269.993330:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077269.993337:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4255 00000100:00080000:1.0:1769077269.993343:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077269.993347:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705920:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077269.993356:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077269.993368:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077269.993392:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616706432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077269.993409:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077269.993413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077269.993421:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077269.993424:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077269.993433:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4255 00000100:00080000:1.0:1769077269.993440:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077269.993442:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077269/real 1769077269] req@ffff995b868c6a00 x1855007616705920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077269.993512:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616705920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077269.993515:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077269.993533:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077269.993537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077269.993547:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077269.993554:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616705920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077269.993563:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616706560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077269.993574:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077269.993631:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077269.993654:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077269/real 1769077269] req@ffff995b868c7100 x1855007616706048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077269.993678:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616706048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077269.993696:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077269.993702:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077269.993707:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077269.993720:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706176:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077269.993777:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077269.993797:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077269/real 1769077269] req@ffff995b868c5500 x1855007616706176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077269.993817:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616706176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077269.993833:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077269.993839:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077269.993844:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706176:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077269.993853:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706304:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077269.993884:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077269.993901:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077269/real 1769077269] req@ffff995b868c4700 x1855007616706304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077269.993928:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616706304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077269.993942:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077269.993946:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077269.993951:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706304:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077269.993982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616706432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077269.994013:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077269.994031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077269/real 1769077269] req@ffff995b868c7480 x1855007616706432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077269.994050:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616706432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077269.994063:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077269.994068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077269.994072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616706432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077269.994081:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616706560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077269.994111:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077269.994129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077269/real 1769077269] req@ffff995b868c4a80 x1855007616706560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077269.994148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616706560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077269.994169:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077269.994174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077269.994179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616706560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077270.696566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077270.696685:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077271.016564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077271.016564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077271.016571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077271.016700:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077271.720582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077271.720832:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077272.040587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077272.040596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077272.040640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077272.040641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077272.744582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077272.744676:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077273.064646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077273.064646:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077273.064740:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077273.064740:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077273.768563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077273.768620:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077274.088585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077274.088589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077274.088601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077274.088608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077274.792575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077274.792640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077275.112580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077275.112580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077275.112581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077275.112584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077275.112634:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077275.112640:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077275.112645:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077275.112655:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4260 00000100:00080000:2.0:1769077275.112685:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077275.112694:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077275.112745:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616706688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077275.112769:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077275.112772:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077275.112774:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077275.112778:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4260 00000100:00080000:2.0:1769077275.112781:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077275.112785:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077275.112796:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077275.112806:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616706816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077275.112817:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616706688:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077275.112818:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077275.112827:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077275.112829:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077275.112833:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4260 00000100:00080000:2.0:1769077275.112835:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077275.112838:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077275.112842:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077275.112854:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616706944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077275.112861:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077275.112864:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077275.112866:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077275.112869:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4260 00000100:00080000:2.0:1769077275.112871:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077275.112874:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077275.112877:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077275.112887:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616707072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077275.112895:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077275.112925:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077275.112931:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077275.112935:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077275.112942:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4260 00000100:00080000:2.0:1769077275.112948:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077275.112953:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077275.112961:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077275.112979:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077275/real 1769077275] req@ffff995b85acb100 x1855007616706688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077275.112995:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616707200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077275.113035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077275.113038:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616706688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077275.113054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077275.113065:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077275.113070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077275.113081:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4260 00000100:00080000:0.0:1769077275.113082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077275.113095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616706688:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077275.113099:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077275.113114:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077275.113126:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077275.113134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077275.113156:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616707328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077275.113176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077275.113177:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077275.113186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077275.113190:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077275.113199:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4260 00000100:00100000:0.0:1769077275.113205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077275/real 1769077275] req@ffff995b85ac8a80 x1855007616706816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077275.113209:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077275.113224:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077275.113240:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077275.113252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616706816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077275.113265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077275.113272:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616707456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077275.113274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077275.113285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077275.113304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077275.113353:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077275.113366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077275/real 1769077275] req@ffff995b85acaa00 x1855007616706944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077275.113396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616706944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077275.113409:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077275.113414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077275.113417:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616706944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077275.113424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707072:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077275.113443:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077275.113454:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077275/real 1769077275] req@ffff995b85aca300 x1855007616707072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077275.113501:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616707072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077275.113527:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077275.113531:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077275.113536:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077275.113543:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707200:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077275.113566:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077275.113580:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077275/real 1769077275] req@ffff995b85acb800 x1855007616707200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077275.113605:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616707200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077275.113617:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077275.113620:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077275.113624:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707200:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077275.113647:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616707328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077275.113705:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077275.113720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077275/real 1769077275] req@ffff995b85ac9c00 x1855007616707328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077275.113737:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616707328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077275.113748:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077275.113751:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077275.113756:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616707328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077275.113764:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616707456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077275.113783:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077275.113794:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077275/real 1769077275] req@ffff995b85ac9880 x1855007616707456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077275.113808:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616707456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077275.113818:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077275.113822:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077275.113825:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616707456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077275.816573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077275.816624:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077276.136544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077276.136547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077276.136550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077276.136577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077276.840587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077276.840674:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077277.160569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077277.160575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077277.160577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077277.160578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077277.864575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077277.864677:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077278.184528:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077278.184565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077278.184574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077278.184574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077278.888585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077278.888808:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077279.208534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077279.208552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077279.208590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077279.208638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077279.912537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077279.912704:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077280.232663:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077280.232667:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077280.232759:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077280.232766:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077280.232834:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077280.232843:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077280.232848:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077280.232867:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4265 00000100:00080000:2.0:1769077280.232882:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077280.232892:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077280.232939:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616707584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077280.232968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077280.232971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077280.232973:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077280.232976:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4265 00000100:00080000:2.0:1769077280.232979:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077280.232983:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077280.232995:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077280.233013:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616707712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077280.233034:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077280.233040:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077280.233044:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077280.233045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616707584:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077280.233069:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4265 00000100:00080000:2.0:1769077280.233076:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077280.233087:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077280.233095:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077280.233115:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616707840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077280.233126:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077280.233128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077280.233130:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077280.233136:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4265 00000100:00080000:2.0:1769077280.233151:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077280.233158:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769077280.233165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077280.233168:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077280.233195:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616707968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077280.233208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077280.233211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077280.233214:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077280.233223:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4265 00000100:00080000:2.0:1769077280.233233:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077280.233239:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077280.233249:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077280.233250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077280/real 1769077280] req@ffff995b85acbb80 x1855007616707584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077280.233283:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616708096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077280.233296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616707584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077280.233303:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077280.233314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077280.233321:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077280.233327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077280.233331:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077280.233339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616707584:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077280.233348:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4265 00000100:00080000:2.0:1769077280.233359:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077280.233365:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077280.233378:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077280.233389:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077280.233401:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616708224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077280.233421:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077280.233426:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077280.233428:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077280.233435:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077280.233454:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4265 00000100:00080000:2.0:1769077280.233465:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077280.233518:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077280.233529:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077280.233530:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077280/real 1769077280] req@ffff995b85ac8380 x1855007616707712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077280.233561:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616708352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077280.233569:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616707712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077280.233623:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077280.233632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077280.233638:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077280.233653:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077280.233694:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077280.233712:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077280/real 1769077280] req@ffff995b85ac9880 x1855007616707840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077280.233728:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616707840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077280.233741:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077280.233746:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077280.233750:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077280.233758:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707968:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077280.233779:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077280.233791:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077280/real 1769077280] req@ffff995b85ac9c00 x1855007616707968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077280.233805:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616707968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077280.233816:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077280.233820:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077280.233824:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616707968:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077280.233831:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708096:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077280.233849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077280.233861:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077280/real 1769077280] req@ffff995b85acb800 x1855007616708096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077280.233874:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616708096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077280.233886:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077280.233889:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077280.233893:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708096:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077280.233916:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616708224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077280.233936:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077280.233948:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077280/real 1769077280] req@ffff995b85aca300 x1855007616708224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077280.233962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616708224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077280.233973:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077280.233977:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077280.233981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616708224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077280.233988:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616708352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077280.234005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077280.234016:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077280/real 1769077280] req@ffff995b85acaa00 x1855007616708352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077280.234029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616708352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077280.234040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077280.234043:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077280.234047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616708352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077280.936629:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077280.936715:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077281.256622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077281.256630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077281.256683:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077281.256711:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077281.960569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077281.960738:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077282.280572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077282.280579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077282.280587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077282.280645:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077282.984554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077282.984702:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077283.304605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077283.304605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077283.304605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077283.304649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077284.008602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077284.008687:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077284.328564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077284.328594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077284.328622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077284.328642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077285.032594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077285.032602:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077285.352562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077285.352576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077285.352583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077285.352614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077285.352624:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1769077285.352628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077285.352629:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077285.352644:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4271 00000100:00080000:2.0:1769077285.352657:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077285.352663:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077285.352703:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616708480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077285.352730:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077285.352733:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077285.352734:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077285.352738:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4271 00000100:00080000:2.0:1769077285.352740:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077285.352745:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077285.352755:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077285.352765:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616708608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077285.352773:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077285.352779:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077285.352781:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077285.352783:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616708480:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077285.352787:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4271 00000100:00080000:2.0:1769077285.352791:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077285.352799:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077285.352806:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077285.352821:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616708736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077285.352828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077285.352831:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077285.352832:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077285.352835:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4271 00000100:00080000:2.0:1769077285.352837:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077285.352841:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077285.352844:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077285.352851:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616708864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077285.352858:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077285.352860:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077285.352863:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077285.352866:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4271 00000800:00000100:1.0:1769077285.352879:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077285.352881:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077285.352886:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077285.352892:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077285.352903:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616708992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077285.352911:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077285.352914:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077285.352915:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077285.352919:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4271 00000100:00080000:2.0:1769077285.352923:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077285.352928:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077285.352932:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077285.352939:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616709120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077285.352945:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077285/real 1769077285] req@ffff995b85ac8a80 x1855007616708480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077285.352946:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077285.352954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077285.352958:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077285.352964:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4271 00000100:00080000:2.0:1769077285.352971:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077285.352977:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077285.352987:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077285.352996:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616709248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077285.353007:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616708480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077285.353021:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077285.353027:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077285.353031:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616708480:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077285.353059:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077285.353094:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077285.353110:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077285/real 1769077285] req@ffff995b85acb100 x1855007616708608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077285.353120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616708608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077285.353126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077285.353129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077285.353131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077285.353136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077285.353151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077285.353158:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077285/real 1769077285] req@ffff995b85acaa00 x1855007616708736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077285.353171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616708736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077285.353178:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077285.353180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077285.353182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077285.353187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708864:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077285.353198:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077285.353213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077285/real 1769077285] req@ffff995b85aca300 x1855007616708864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077285.353221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616708864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077285.353228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077285.353230:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077285.353233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708864:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077285.353237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708992:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077285.353247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077285.353253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077285/real 1769077285] req@ffff995b85acb800 x1855007616708992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077285.353261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616708992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077285.353271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077285.353273:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077285.353275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616708992:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077285.353279:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616709120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077285.353289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077285.353295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077285/real 1769077285] req@ffff995b85ac9c00 x1855007616709120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077285.353302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616709120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077285.353308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077285.353310:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077285.353312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616709120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077285.353315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616709248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077285.353326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077285.353333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077285/real 1769077285] req@ffff995b85ac9880 x1855007616709248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077285.353339:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616709248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077285.353344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077285.353346:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077285.353349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616709248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077286.056596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077286.056681:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077286.376581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077286.376585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077286.376597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077286.376606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077287.080525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077287.080589:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077287.400552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077287.400570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077287.400577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077287.400595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077288.104606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077288.104663:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077288.424627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077288.424641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077288.424642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077288.424675:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077289.128591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077289.128629:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077289.448568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077289.448575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077289.448575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077289.448576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077290.152561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077290.152612:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077290.472584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077290.472585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077290.472596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077290.472640:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077290.472646:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077290.472652:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077290.472657:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077290.472671:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4276 00000100:00080000:0.0:1769077290.472685:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077290.472692:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077290.472742:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616709376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077290.472766:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077290.472768:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077290.472770:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077290.472773:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4276 00000100:00080000:0.0:1769077290.472775:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077290.472779:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077290.472788:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077290.472804:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616709504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077290.472819:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616709376:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077290.472822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077290.472829:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077290.472831:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077290.472837:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4276 00000100:00080000:0.0:1769077290.472846:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077290.472852:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077290.472860:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077290.472875:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616709632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077290.472890:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077290.472894:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077290.472901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077290.472904:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077290.472909:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4276 00000100:00080000:0.0:1769077290.472914:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077290.472919:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077290.472923:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077290.472940:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616709760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077290.472948:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077290/real 1769077290] req@ffff995b84643800 x1855007616709376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077290.472955:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077290.472970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077290.472976:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077290.472981:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4276 00000100:00080000:0.0:1769077290.472985:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077290.472995:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077290.472995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616709376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077290.473011:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769077290.473018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077290.473027:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616709888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077290.473028:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077290.473044:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616709376:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077290.473051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077290.473055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077290.473058:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077290.473062:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4276 00000100:00080000:0.0:1769077290.473066:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077290.473069:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077290.473073:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077290.473084:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616709504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077290.473085:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616710016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077290.473109:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077290.473111:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077290.473113:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077290.473117:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4276 00000100:00080000:0.0:1769077290.473121:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077290.473124:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077290.473127:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077290.473137:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616710144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077290.473140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077290.473170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077290/real 1769077290] req@ffff995b84641180 x1855007616709504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077290.473187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616709504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077290.473193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077290.473196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077290.473198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616709504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077290.473203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616709632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077290.473217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077290.473224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077290/real 1769077290] req@ffff995b84641500 x1855007616709632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077290.473232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616709632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077290.473239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077290.473242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077290.473244:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616709632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077290.473248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616709760:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077290.473259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077290.473264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077290/real 1769077290] req@ffff995b84642680 x1855007616709760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077290.473272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616709760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077290.473282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077290.473285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077290.473286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616709760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077290.473290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616709888:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077290.473299:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077290.473304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077290/real 1769077290] req@ffff995b84640e00 x1855007616709888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077290.473311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616709888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077290.473318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077290.473320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077290.473322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616709888:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077290.473338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616710016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077290.473349:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077290.473355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077290/real 1769077290] req@ffff995b84641c00 x1855007616710016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077290.473362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616710016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077290.473368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077290.473370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077290.473372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616710016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077290.473376:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616710144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077290.473389:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077290.473399:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077290/real 1769077290] req@ffff995b84640a80 x1855007616710144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077290.473443:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616710144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077290.473472:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077290.473475:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077290.473478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616710144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077291.176567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077291.176641:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077291.496570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077291.496595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077291.496605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077291.496614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077292.200562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077292.200623:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077292.520645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077292.520669:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077292.520669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077292.520680:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077293.224529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077293.224620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077293.544534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077293.544534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077293.544549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077293.544566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077294.248538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077294.248600:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077294.568539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077294.568539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077294.568555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077294.568585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769077295.272680:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077295.272727:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077295.592562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077295.592562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077295.592572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077295.592598:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077295.592604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077295.592608:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077295.592617:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4281 00000100:00100000:3.0:1769077295.592618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077295.592629:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077295.592637:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077295.592676:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616710272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077295.592700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077295.592702:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077295.592704:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077295.592707:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4281 00000100:00080000:1.0:1769077295.592709:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077295.592713:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077295.592722:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077295.592732:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616710272:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077295.592734:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616710400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077295.592746:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077295.592748:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077295.592749:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077295.592752:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4281 00000100:00080000:1.0:1769077295.592754:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077295.592757:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077295.592761:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077295.592769:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616710528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077295.592776:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077295.592779:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077295.592780:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077295.592783:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4281 00000100:00080000:1.0:1769077295.592801:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077295.592805:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077295.592810:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769077295.592813:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077295.592826:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616710656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077295.592833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077295.592835:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077295.592837:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077295.592839:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4281 00000100:00080000:1.0:1769077295.592843:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077295.592846:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077295.592850:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077295.592854:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077295/real 1769077295] req@ffff995b868c7800 x1855007616710272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077295.592862:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616710784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077295.592890:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077295.592894:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616710272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077295.592895:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077295.592898:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077295.592906:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4281 00000100:00080000:0.0:1769077295.592911:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077295.592911:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077295.592915:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769077295.592918:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769077295.592922:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077295.592923:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616710272:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077295.592938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616710912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077295.592951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616710400:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077295.592952:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077295.592958:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077295.592960:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077295.592964:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4281 00000100:00080000:1.0:1769077295.592971:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077295.592975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077295.592978:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077295.592983:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077295.592991:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077295/real 1769077295] req@ffff995b868c6d80 x1855007616710400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077295.592999:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616711040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077295.593004:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616710400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077295.593024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077295.593028:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077295.593031:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616710400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077295.593036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616710528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077295.593050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077295.593056:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077295/real 1769077295] req@ffff995b868c6300 x1855007616710528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077295.593064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616710528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077295.593075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077295.593077:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077295.593079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616710528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077295.593083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616710656:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077295.593094:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077295.593099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077295/real 1769077295] req@ffff995b868c4a80 x1855007616710656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077295.593106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616710656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077295.593112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077295.593114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077295.593116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616710656:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077295.593120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616710784:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077295.593139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077295.593150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077295/real 1769077295] req@ffff995b868c7480 x1855007616710784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077295.593163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616710784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077295.593173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077295.593176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077295.593181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616710784:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077295.593199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616710912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077295.593215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077295.593223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077295/real 1769077295] req@ffff995b868c4700 x1855007616710912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077295.593231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616710912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077295.593237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077295.593239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077295.593241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616710912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077295.593245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616711040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077295.593254:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077295.593260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077295/real 1769077295] req@ffff995b868c5500 x1855007616711040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077295.593267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616711040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077295.593273:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077295.593274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077295.593276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616711040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077296.296533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077296.296635:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077296.616623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077296.616623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077296.616623:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077296.616626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077297.320547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077297.320640:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077297.640546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077297.640547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077297.640568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077297.640586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077298.344536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077298.344597:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077298.664542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077298.664542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077298.664574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077298.664611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077299.368572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077299.368614:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077299.688536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077299.688539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077299.688587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077299.688609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077300.392601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077300.392684:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077300.712574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077300.712609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077300.712611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077300.712615:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077300.712624:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769077300.712629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077300.712631:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077300.712648:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4286 00000100:00080000:2.0:1769077300.712663:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077300.712673:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077300.712717:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616711168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077300.712748:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077300.712752:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077300.712754:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077300.712757:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4286 00000100:00080000:2.0:1769077300.712761:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077300.712765:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077300.712775:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077300.712786:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616711296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077300.712795:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077300.712798:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077300.712800:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077300.712804:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4286 00000100:00080000:2.0:1769077300.712806:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077300.712814:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616711168:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077300.712814:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077300.712820:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077300.712833:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616711424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077300.712841:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077300.712844:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077300.712845:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077300.712848:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4286 00000100:00080000:2.0:1769077300.712850:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077300.712854:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077300.712857:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077300.712865:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616711552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077300.712870:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077300.712873:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077300.712874:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077300.712878:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4286 00000100:00080000:2.0:1769077300.712881:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077300.712883:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077300.712886:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077300.712891:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616711680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077300.712897:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077300.712899:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077300.712901:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077300.712903:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4286 00000100:00080000:2.0:1769077300.712906:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077300.712913:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077300.712915:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077300.712920:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077300.712929:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616711808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077300.712937:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077300.712940:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077300.712941:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077300.712944:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4286 00000100:00080000:2.0:1769077300.712946:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077300.712949:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077300.712952:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077300.712960:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616711936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077300.712973:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077300/real 1769077300] req@ffff995b85ac8380 x1855007616711168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077300.712997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616711168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077300.713010:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077300.713015:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077300.713020:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616711168:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077300.713048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616711296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077300.713077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077300.713091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077300/real 1769077300] req@ffff995b85acbb80 x1855007616711296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077300.713104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616711296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077300.713113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077300.713117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077300.713120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616711296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077300.713128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616711424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077300.713150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077300.713162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077300/real 1769077300] req@ffff995b85ac9880 x1855007616711424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077300.713175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616711424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077300.713186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077300.713190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077300.713193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616711424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077300.713201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616711552:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077300.713223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077300.713274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077300/real 1769077300] req@ffff995b85ac9c00 x1855007616711552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077300.713289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616711552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077300.713299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077300.713304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077300.713307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616711552:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077300.713315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616711680:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077300.713335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077300.713346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077300/real 1769077300] req@ffff995b85acb800 x1855007616711680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077300.713366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616711680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077300.713376:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077300.713380:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077300.713383:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616711680:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077300.713391:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616711808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077300.713410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077300.713421:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077300/real 1769077300] req@ffff995b85aca300 x1855007616711808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077300.713434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616711808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077300.713443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077300.713447:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077300.713450:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616711808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077300.713483:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616711936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077300.713505:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077300.713518:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077300/real 1769077300] req@ffff995b85acaa00 x1855007616711936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077300.713530:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616711936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077300.713545:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077300.713548:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077300.713552:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616711936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077301.416602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077301.416628:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077301.736543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077301.736590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077301.736604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077301.736621:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077302.440532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077302.440606:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077302.760583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077302.760583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077302.760588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077302.760620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077303.464567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077303.464659:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077303.784540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077303.784559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077303.784564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077303.784595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077304.488526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077304.488579:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077304.808531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077304.808535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077304.808551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077304.808567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077305.512567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077305.512624:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077305.832570:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077305.832574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077305.832575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077305.832575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077305.832585:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077305.832591:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077305.832602:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4291 00000100:00080000:3.0:1769077305.832615:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077305.832620:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077305.832626:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077305.832670:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616712064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077305.832698:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077305.832702:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077305.832704:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077305.832708:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4291 00000100:00080000:3.0:1769077305.832713:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077305.832717:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077305.832727:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077305.832740:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616712192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077305.832745:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616712064:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077305.832758:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077305.832763:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077305.832765:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077305.832769:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4291 00000100:00080000:3.0:1769077305.832774:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077305.832778:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077305.832783:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077305.832796:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616712320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077305.832807:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077305.832810:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077305.832812:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077305.832815:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4291 00000100:00080000:3.0:1769077305.832819:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077305.832823:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077305.832828:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769077305.832831:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077305.832841:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616712448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077305.832853:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077305.832857:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077305.832859:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077305.832864:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4291 00000100:00080000:3.0:1769077305.832868:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077305.832873:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077305.832878:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077305/real 1769077305] req@ffff995b8455bb80 x1855007616712064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077305.832880:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077305.832896:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616712576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077305.832908:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077305.832913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769077305.832914:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616712064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077305.832917:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077305.832927:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4291 00000100:00080000:3.0:1769077305.832933:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077305.832938:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077305.832939:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077305.832947:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077305.832949:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077305.832955:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616712064:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077305.832961:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616712704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077305.832971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077305.832975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077305.832977:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077305.832982:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4291 00000100:00080000:3.0:1769077305.832986:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077305.832987:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616712192:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077305.832993:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077305.832999:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077305.833012:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616712832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077305.833031:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077305.833048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077305/real 1769077305] req@ffff995b84559500 x1855007616712192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077305.833062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616712192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077305.833078:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077305.833081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077305.833086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616712192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077305.833092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616712320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077305.833110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077305.833121:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077305/real 1769077305] req@ffff995b84558a80 x1855007616712320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077305.833134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616712320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077305.833145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077305.833161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077305.833165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616712320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077305.833172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616712448:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077305.833192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077305.833203:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077305/real 1769077305] req@ffff995b8455a680 x1855007616712448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077305.833215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616712448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077305.833225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077305.833228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077305.833232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616712448:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077305.833239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616712576:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077305.833255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077305.833266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077305/real 1769077305] req@ffff995b8455b100 x1855007616712576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077305.833278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616712576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077305.833288:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077305.833291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077305.833294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616712576:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077305.833300:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616712704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077305.833317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077305.833328:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077305/real 1769077305] req@ffff995b8455aa00 x1855007616712704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077305.833341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616712704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077305.833350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077305.833353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077305.833357:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616712704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077305.833376:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616712832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077305.833405:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077305.833417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077305/real 1769077305] req@ffff995b84558000 x1855007616712832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077305.833430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616712832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077305.833440:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077305.833443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077305.833447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616712832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077306.536529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077306.536605:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077306.856611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077306.856617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077306.856631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077306.856661:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077307.560603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077307.560908:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077307.880592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077307.880606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077307.880630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077307.880664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077308.584542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077308.584560:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077308.904552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077308.904556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077308.904564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077308.904628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077309.608548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077309.608614:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077309.928574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077309.928574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077309.928574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077309.928609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077310.632564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077310.632629:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077310.952559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077310.952560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077310.952570:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769077310.952572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077310.952575:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077310.952585:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4296 00000100:00080000:3.0:1769077310.952597:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077310.952604:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077310.952609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077310.952625:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077310.952643:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616712960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077310.952666:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077310.952669:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077310.952670:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077310.952673:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4296 00000100:00080000:3.0:1769077310.952675:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077310.952679:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077310.952688:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077310.952697:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616713088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077310.952705:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077310.952709:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769077310.952711:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616712960:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077310.952711:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077310.952716:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4296 00000100:00080000:3.0:1769077310.952722:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077310.952725:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077310.952729:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077310.952737:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616713216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077310.952748:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077310.952751:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077310.952752:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077310.952754:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4296 00000100:00080000:3.0:1769077310.952757:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077310.952759:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077310.952762:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077310.952769:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616713344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077310.952774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077310.952776:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077310.952778:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769077310.952783:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077310.952786:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4296 00000100:00080000:3.0:1769077310.952791:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077310.952793:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077310.952797:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077310.952805:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616713472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077310.952811:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077310.952814:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077310.952815:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077310.952819:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4296 00000100:00080000:3.0:1769077310.952823:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077310.952824:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077310/real 1769077310] req@ffff995b8455ad80 x1855007616712960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077310.952828:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077310.952834:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077310.952848:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616713600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077310.952852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616712960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077310.952868:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077310.952872:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077310.952872:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077310.952874:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077310.952881:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077310.952882:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4296 00000100:00080000:3.0:1769077310.952889:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077310.952891:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616712960:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077310.952894:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077310.952900:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077310.952909:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616713728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077310.952922:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077310.952943:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077310.952952:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077310/real 1769077310] req@ffff995b8455b480 x1855007616713088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077310.952962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616713088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077310.952969:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077310.952971:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077310.952973:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077310.952977:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077310.952988:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077310.952994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077310/real 1769077310] req@ffff995b84558700 x1855007616713216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077310.953002:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616713216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077310.953009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077310.953011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077310.953013:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077310.953017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713344:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077310.953027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077310.953033:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077310/real 1769077310] req@ffff995b8455a300 x1855007616713344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077310.953041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616713344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077310.953051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077310.953053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077310.953055:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713344:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077310.953058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713472:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077310.953096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077310.953104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077310/real 1769077310] req@ffff995b84559c00 x1855007616713472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077310.953111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616713472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077310.953117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077310.953119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077310.953122:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713472:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077310.953126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616713600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077310.953143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077310.953154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077310/real 1769077310] req@ffff995b84559880 x1855007616713600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077310.953168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616713600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077310.953177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077310.953181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077310.953185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616713600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077310.953192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616713728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077310.953208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077310.953215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077310/real 1769077310] req@ffff995b84558000 x1855007616713728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077310.953223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616713728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077310.953228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077310.953230:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077310.953232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616713728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077311.656558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077311.656630:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077311.976564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077311.976591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077311.976597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077311.976630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077312.680592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077312.680611:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077313.000548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077313.000589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077313.000591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077313.000632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077313.704535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077313.704582:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077314.024560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077314.024561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077314.024576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077314.024597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077314.728558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077314.728623:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077315.048540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077315.048541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077315.048541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077315.048550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077315.752528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077315.752653:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077316.072527:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077316.072552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077316.072560:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077316.072565:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077316.072575:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4301 00000100:00080000:3.0:1769077316.072586:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077316.072593:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077316.072596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077316.072600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077316.072633:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616713856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077316.072640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077316.072665:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077316.072669:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077316.072672:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077316.072677:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4301 00000100:00080000:3.0:1769077316.072681:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077316.072686:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077316.072697:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077316.072702:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616713856:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077316.072715:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616713984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077316.072727:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077316.072739:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077316.072741:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077316.072746:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4301 00000100:00080000:3.0:1769077316.072757:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077316.072762:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077316.072768:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077316.072771:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077316.072798:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616714112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077316.072810:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077316/real 1769077316] req@ffff995b8455aa00 x1855007616713856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077316.072829:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616713856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077316.072839:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077316.072843:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077316.072848:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616713856:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077316.072863:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077316.072868:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1769077316.072871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713984:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077316.072872:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077316.072881:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4301 00000100:00080000:3.0:1769077316.072887:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077316.072892:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769077316.072899:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077316.072911:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077316/real 1769077316] req@ffff995b8455b100 x1855007616713984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077316.072924:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616713984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077316.072932:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077316.072935:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077316.072938:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616713984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077316.072944:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616714112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077316.072962:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077316.072973:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077316/real 1769077316] req@ffff995b8455a680 x1855007616714112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077316.073000:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616714112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077316.073011:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077316.073016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077316.073020:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616714112:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769077316.073023:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077316.073081:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616714240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077316.073099:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077316.073103:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077316.073106:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077316.073111:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4301 00000100:00080000:3.0:1769077316.073117:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077316.073122:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077316.073127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616714240:192.168.204.132@tcp:8:kworker.0 00000008:00100000:3.0:1769077316.073128:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077316.073147:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616714368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077316.073159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077316.073159:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077316.073166:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077316.073168:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077316.073175:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4301 00000100:00100000:2.0:1769077316.073180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077316/real 1769077316] req@ffff995b84558a80 x1855007616714240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077316.073181:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077316.073190:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077316.073198:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077316.073213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616714240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077316.073215:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616714496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077316.073238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077316.073246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077316.073253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616714240:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077316.073253:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077316.073262:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077316.073265:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077316.073271:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4301 00000100:00080000:3.0:1769077316.073275:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077316.073279:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077316.073283:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616714368:192.168.204.132@tcp:8:kworker.0 00000008:00100000:3.0:1769077316.073286:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077316.073303:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616714624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077316.073313:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077316.073329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077316/real 1769077316] req@ffff995b84559500 x1855007616714368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077316.073342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616714368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077316.073350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077316.073353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077316.073356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616714368:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077316.073362:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616714496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077316.073380:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077316.073390:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077316/real 1769077316] req@ffff995b8455bb80 x1855007616714496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077316.073402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616714496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077316.073415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077316.073418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077316.073422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616714496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077316.073435:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616714624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077316.073488:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077316.073503:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077316/real 1769077316] req@ffff995b84558a80 x1855007616714624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077316.073516:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616714624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077316.073524:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077316.073527:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077316.073530:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616714624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077316.776557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077316.776625:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077317.096548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077317.096548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077317.096548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077317.096596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077317.800527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077317.800583:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077318.120560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077318.120560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077318.120563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077318.120566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077318.824568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077318.824613:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077319.144544:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077319.144544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077319.144546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077319.144549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077319.848540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077319.848594:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077320.168509:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077320.168542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077320.168582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077320.168587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077320.872550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077320.872602:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077321.192559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077321.192568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077321.192570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077321.192579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077321.192585:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077321.192595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077321.192596:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4306 00000100:00080000:0.0:1769077321.192609:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077321.192609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077321.192618:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077321.192653:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616714752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077321.192674:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077321.192677:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077321.192679:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077321.192683:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4306 00000100:00080000:0.0:1769077321.192687:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077321.192691:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077321.192700:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077321.192709:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616714752:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077321.192714:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616714880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077321.192726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077321.192728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077321.192730:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077321.192732:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4306 00000100:00080000:0.0:1769077321.192735:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077321.192738:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077321.192741:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077321.192749:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616715008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077321.192760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077321.192762:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077321.192763:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077321.192766:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4306 00000100:00080000:0.0:1769077321.192769:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077321.192772:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077321.192775:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077321.192781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077321.192787:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616715136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077321.192796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077321.192800:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077321.192802:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077321.192805:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4306 00000100:00080000:0.0:1769077321.192809:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077321.192814:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077321.192818:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077321.192824:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077321/real 1769077321] req@ffff995b84640000 x1855007616714752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077321.192831:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616715264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077321.192844:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077321.192850:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077321.192852:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077321.192857:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4306 00000100:00100000:2.0:1769077321.192862:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616714752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077321.192864:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077321.192872:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077321.192879:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077321.192883:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077321.192908:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077321.192909:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616715392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077321.192916:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616714752:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077321.192920:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077321.192925:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077321.192926:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077321.192930:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4306 00000100:00080000:0.0:1769077321.192933:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077321.192936:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077321.192939:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077321.192948:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616714880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077321.192950:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616715520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077321.192983:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077321.192997:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077321/real 1769077321] req@ffff995b84641f80 x1855007616714880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077321.193010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616714880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077321.193027:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077321.193031:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077321.193035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616714880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077321.193042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077321.193062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077321.193074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077321/real 1769077321] req@ffff995b84640380 x1855007616715008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077321.193093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616715008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077321.193104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077321.193107:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077321.193110:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077321.193117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715136:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077321.193136:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077321.193146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077321/real 1769077321] req@ffff995b84640a80 x1855007616715136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077321.193158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616715136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077321.193168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077321.193171:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077321.193174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715136:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077321.193180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715264:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077321.193198:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077321.193208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077321/real 1769077321] req@ffff995b84641c00 x1855007616715264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077321.193219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616715264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077321.193234:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077321.193237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077321.193241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077321.193246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616715392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077321.193265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077321.193273:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077321/real 1769077321] req@ffff995b84640e00 x1855007616715392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077321.193280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616715392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077321.193285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077321.193287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077321.193289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616715392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077321.193302:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616715520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077321.193312:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077321.193317:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077321/real 1769077321] req@ffff995b84642680 x1855007616715520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077321.193324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616715520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077321.193329:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077321.193331:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077321.193333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616715520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077321.896568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077321.896620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077322.216588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077322.216591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077322.216593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077322.216596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077322.920547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077322.920602:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077323.240517:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077323.240534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077323.240653:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077323.240702:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077323.944529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077323.944587:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077324.264554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077324.264555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077324.264557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077324.264664:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077324.968530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077324.969547:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077325.288557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077325.288557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077325.288557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077325.288558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077325.992529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077325.992566:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077326.312531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077326.312531:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077326.312534:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077326.312537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077326.312566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077326.312573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077326.312577:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077326.312585:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4312 00000100:00080000:1.0:1769077326.312593:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077326.312599:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077326.312632:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616715648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077326.312651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077326.312654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077326.312655:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077326.312658:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4312 00000100:00080000:1.0:1769077326.312661:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077326.312664:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077326.312672:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077326.312682:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616715648:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077326.312688:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616715776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077326.312697:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077326.312699:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077326.312701:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077326.312704:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4312 00000100:00080000:1.0:1769077326.312706:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077326.312708:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077326.312711:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077326.312720:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616715904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077326.312725:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077326.312728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077326.312728:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077326.312730:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4312 00000100:00080000:1.0:1769077326.312734:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077326.312736:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077326.312739:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077326.312746:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616716032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077326.312753:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077326.312754:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077326.312758:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077326.312759:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077326.312763:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4312 00000100:00080000:1.0:1769077326.312765:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077326.312767:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077326.312770:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077326.312777:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616716160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077326.312783:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077326.312785:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077326.312787:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077326.312790:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077326/real 1769077326] req@ffff995b868c7100 x1855007616715648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077326.312793:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4312 00000100:00080000:1.0:1769077326.312798:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077326.312829:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077326.312833:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616715648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077326.312837:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077326.312850:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077326.312855:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616716288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077326.312859:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077326.312866:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616715648:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077326.312872:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077326.312876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077326.312878:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077326.312884:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4312 00000100:00080000:1.0:1769077326.312888:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077326.312894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715776:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077326.312895:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077326.312912:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077326.312920:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077326.312926:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616716416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077326.312936:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077326/real 1769077326] req@ffff995b868c6a00 x1855007616715776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077326.312951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616715776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077326.312961:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077326.312965:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077326.312969:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077326.312973:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077326.312988:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077326.312995:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077326/real 1769077326] req@ffff995b868c5c00 x1855007616715904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077326.313003:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616715904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077326.313009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077326.313012:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077326.313014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616715904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077326.313017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716032:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077326.313027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077326.313032:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077326/real 1769077326] req@ffff995b868c5500 x1855007616716032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077326.313039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616716032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077326.313044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077326.313045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077326.313047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716032:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077326.313051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077326.313059:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077326.313064:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077326/real 1769077326] req@ffff995b868c4700 x1855007616716160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077326.313070:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616716160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077326.313076:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077326.313078:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077326.313079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077326.313082:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616716288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077326.313091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077326.313096:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077326/real 1769077326] req@ffff995b868c7480 x1855007616716288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077326.313106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616716288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077326.313114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077326.313117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077326.313120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616716288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077326.313137:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616716416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077326.313155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077326.313163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077326/real 1769077326] req@ffff995b868c4a80 x1855007616716416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077326.313170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616716416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077326.313175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077326.313176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077326.313178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616716416:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769077327.016640:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077327.016694:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077327.336599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077327.336599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077327.336599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077327.336599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077328.040559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077328.040640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077328.360577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077328.360589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077328.360595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077328.360627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077329.064551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077329.064608:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077329.384591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077329.384591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077329.384591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077329.384592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077330.088522:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077330.088585:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077330.408521:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077330.408529:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077330.408540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077330.408567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077331.112518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077331.112571:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077331.432542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077331.432542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077331.432562:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077331.432570:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769077331.432574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077331.432575:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077331.432582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077331.432588:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4317 00000100:00080000:1.0:1769077331.432599:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077331.432606:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077331.432643:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616716544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077331.432668:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077331.432671:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077331.432672:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077331.432676:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4317 00000100:00080000:1.0:1769077331.432680:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077331.432684:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077331.432693:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077331.432703:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616716544:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077331.432707:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616716672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077331.432740:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077331.432744:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077331.432745:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077331.432750:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4317 00000100:00080000:1.0:1769077331.432754:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077331.432758:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077331.432762:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077331.432774:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616716800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077331.432783:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077331.432787:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077331.432789:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077331.432793:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4317 00000100:00080000:1.0:1769077331.432797:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077331.432801:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077331.432806:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077331.432810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077331.432821:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616716928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077331.432832:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077331.432835:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077331.432837:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077331.432842:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716800:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077331.432843:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4317 00000100:00080000:1.0:1769077331.432850:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077331.432856:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077331.432861:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077331.432873:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616717056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077331.432879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716928:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769077331.432890:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077331.432894:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077331.432896:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077331.432901:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4317 00000100:00100000:2.0:1769077331.432905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717056:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769077331.432906:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077331.432914:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077331.432919:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077331.432931:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616717184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077331.432945:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077331.432949:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077331.432951:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077331.432955:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4317 00000100:00100000:2.0:1769077331.432958:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616717184:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077331.432960:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077331.432967:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077331.432972:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077331.432983:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616717312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077331.433011:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616717312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077332.136548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077332.136607:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077332.456602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077332.456602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077332.456627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077332.456648:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077333.160541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077333.160599:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077333.480575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077333.480575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077333.480591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077333.480613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077334.184555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077334.184618:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077334.504555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077334.504556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077334.504581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077334.504709:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:1.0:1769077334.504798:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769077334.504846:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077334.504868:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077334.504914:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077334.504925:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077334.504934:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077334.504942:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077334.504950:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077334.504957:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769077334.505034:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077331/real 1769077334] req@ffff995b868c5500 x1855007616717312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077334.505061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616717312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077334.505075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077334.505081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077334.505086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616717312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077334.505095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077331/real 1769077334] req@ffff995b868c4700 x1855007616717184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077334.505109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616717184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077334.505120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077334.505125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077334.505129:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616717184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077334.505137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077331/real 1769077334] req@ffff995b868c7480 x1855007616717056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077334.505159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616717056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077334.505173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077334.505177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077334.505181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077334.505188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077331/real 1769077334] req@ffff995b868c4a80 x1855007616716928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077334.505201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616716928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077334.505210:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077334.505214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077334.505218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716928:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077334.505225:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077331/real 1769077334] req@ffff995b868c6d80 x1855007616716672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077334.505247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616716672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077334.505256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077334.505260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077334.505263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077334.505270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077331/real 1769077334] req@ffff995b868c7800 x1855007616716800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077334.505285:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616716800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077334.505299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077334.505303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077334.505307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616716800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077334.505316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077331/real 1769077334] req@ffff995b868c6300 x1855007616716544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077334.505331:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616716544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077334.505340:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077334.505344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077334.505349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616716544:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077335.208522:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077335.208597:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077335.528566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077335.528570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077335.528586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077335.528619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077336.232578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077336.232661:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077336.552601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077336.552602:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077336.552608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077336.552614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077336.552621:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077336.552650:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4322 00000100:00080000:1.0:1769077336.552696:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077336.552703:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077336.552704:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077336.552706:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077336.552746:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616717440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077336.552770:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077336.552773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077336.552774:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077336.552777:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4322 00000100:00080000:1.0:1769077336.552780:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077336.552782:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077336.552790:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077336.552802:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616717568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077336.552815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616717440:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077336.552817:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077336.552824:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077336.552828:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077336.552833:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4322 00000100:00080000:1.0:1769077336.552837:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077336.552841:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077336.552847:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077336.552859:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616717696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077336.552871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077336.552875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077336.552877:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077336.552882:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4322 00000100:00080000:1.0:1769077336.552889:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077336.552894:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077336.552900:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077336.552902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077336.552916:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616717824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077336.552927:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077336.552931:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077336.552933:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077336.552938:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4322 00000100:00080000:1.0:1769077336.552943:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077336.552950:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077336/real 1769077336] req@ffff995b868c5c00 x1855007616717440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077336.552951:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077336.552962:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077336.552980:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616717952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077336.552984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616717440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077336.553003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077336.553009:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077336.553010:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077336.553012:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077336.553021:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077336.553023:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4322 00000100:00080000:1.0:1769077336.553030:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077336.553032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616717440:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077336.553038:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077336.553046:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077336.553058:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616718080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077336.553068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717568:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077336.553073:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077336.553078:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077336.553081:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077336.553087:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4322 00000100:00080000:1.0:1769077336.553093:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077336.553099:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769077336.553105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769077336.553107:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077336.553124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077336/real 1769077336] req@ffff995b868c6a00 x1855007616717568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077336.553138:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616718208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077336.553145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616717568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077336.553159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077336.553164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077336.553169:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077336.553178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077336.553203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077336.553216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077336/real 1769077336] req@ffff995b868c7100 x1855007616717696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077336.553228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616717696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077336.553241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077336.553244:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077336.553248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077336.553256:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717824:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077336.553278:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077336.553290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077336/real 1769077336] req@ffff995b868c6300 x1855007616717824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077336.553304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616717824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077336.553315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077336.553319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077336.553322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717824:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077336.553330:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717952:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077336.553351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077336.553362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077336/real 1769077336] req@ffff995b868c7800 x1855007616717952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077336.553374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616717952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077336.553385:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077336.553389:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077336.553393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616717952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077336.553415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616718080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077336.553439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077336.553476:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077336/real 1769077336] req@ffff995b868c6d80 x1855007616718080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077336.553493:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616718080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077336.553503:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077336.553506:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077336.553510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616718080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077336.553518:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616718208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077336.553541:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077336.553555:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077336/real 1769077336] req@ffff995b868c4a80 x1855007616718208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077336.553571:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616718208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077336.553580:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077336.553584:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077336.553588:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616718208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077337.256532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077337.256582:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077337.576573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077337.576573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077337.576656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077337.576719:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077338.280620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077338.280686:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077338.600574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077338.600575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077338.600627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077338.600630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077339.304577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077339.304628:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077339.624594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077339.624594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077339.624640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077339.624642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077340.328554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077340.328648:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077340.648605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077340.648612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077340.648623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077340.648655:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077341.352581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077341.352654:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077341.672600:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077341.672600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077341.672609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077341.672609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077341.672612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077341.672622:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077341.672636:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4327 00000100:00080000:2.0:1769077341.672647:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077341.672655:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077341.672657:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077341.672699:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616718336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077341.672726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077341.672730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077341.672731:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077341.672734:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4327 00000100:00080000:2.0:1769077341.672736:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077341.672739:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077341.672748:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077341.672760:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616718464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077341.672769:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077341.672771:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616718336:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077341.672776:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077341.672784:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077341.672788:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4327 00000100:00080000:2.0:1769077341.672790:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077341.672793:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077341.672796:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077341.672811:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616718592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077341.672819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077341.672821:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077341.672823:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077341.672826:0:37:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4327 00000100:00080000:2.0:1769077341.672828:0:37:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077341.672830:0:37:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077341.672834:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077341.672841:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616718720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077341.672847:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077341.672850:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077341.672852:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077341.672854:0:37:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4327 00000100:00080000:2.0:1769077341.672857:0:37:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077341.672859:0:37:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077341.672862:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769077341.672864:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077341.672877:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616718848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077341.672886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077341.672890:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077341.672892:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077341.672895:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4327 00000100:00080000:2.0:1769077341.672898:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077341.672901:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077341.672906:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077341.672925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077341/real 1769077341] req@ffff995b85acb100 x1855007616718336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077341.672926:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616718976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077341.672969:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077341.672978:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077341.672981:0:37:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077341.672987:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4327 00000100:00100000:0.0:1769077341.672990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616718336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077341.672992:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077341.673005:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077341.673020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077341.673023:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077341.673033:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077341.673034:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616719104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077341.673047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616718336:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077341.673086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616718464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077341.673105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077341.673114:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077341/real 1769077341] req@ffff995b85ac8a80 x1855007616718464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077341.673128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616718464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077341.673134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077341.673136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077341.673138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616718464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077341.673142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616718592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077341.673152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077341.673158:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077341/real 1769077341] req@ffff995b85acaa00 x1855007616718592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077341.673166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616718592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077341.673173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077341.673175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077341.673177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616718592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077341.673180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616718720:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077341.673193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077341.673199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077341/real 1769077341] req@ffff995b85aca300 x1855007616718720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077341.673206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616718720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077341.673217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077341.673219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077341.673221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616718720:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077341.673225:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616718848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077341.673234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077341.673240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077341/real 1769077341] req@ffff995b85acb800 x1855007616718848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077341.673247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616718848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077341.673254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077341.673256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077341.673257:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616718848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077341.673261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616718976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077341.673271:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077341.673277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077341/real 1769077341] req@ffff995b85ac9c00 x1855007616718976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077341.673283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616718976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077341.673288:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077341.673290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077341.673293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616718976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077341.673297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616719104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077341.673306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077341.673311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077341/real 1769077341] req@ffff995b85ac9880 x1855007616719104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077341.673318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616719104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077341.673323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077341.673325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077341.673327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616719104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077342.376586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077342.376660:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077342.696597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077342.696597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077342.696609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077342.696660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077343.400696:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077343.400702:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077343.720560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077343.720574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077343.720591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077343.720604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077344.424592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077344.424672:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077344.744533:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077344.744570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077344.744594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077344.744616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077345.448535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077345.448650:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077345.768552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077345.768616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077345.768622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077345.768651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077346.472599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077346.472621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077346.792604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077346.792611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077346.792650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077346.792660:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077346.792668:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077346.792674:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077346.792682:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077346.792687:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4332 00000100:00080000:3.0:1769077346.792703:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077346.792712:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077346.792752:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616719232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077346.792792:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077346.792796:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077346.792798:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077346.792803:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4332 00000100:00080000:3.0:1769077346.792808:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077346.792815:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077346.792826:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077346.792837:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616719232:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077346.792839:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616719360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077346.792863:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077346.792865:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077346.792867:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077346.792870:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4332 00000100:00080000:3.0:1769077346.792873:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077346.792876:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077346.792881:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077346.792892:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616719488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077346.792900:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077346.792903:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077346.792904:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077346.792906:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4332 00000100:00080000:3.0:1769077346.792908:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077346.792910:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077346.792914:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077346.792920:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616719616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077346.792926:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077346.792930:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077346.792932:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077346.792935:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4332 00000100:00080000:3.0:1769077346.792937:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077346.792939:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077346.792942:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769077346.792949:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077346.792957:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616719744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077346.792969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077346.792972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077346.792974:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077346.792977:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4332 00000100:00080000:3.0:1769077346.792980:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077346.792983:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077346.792987:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077346.792994:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616719872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077346.793001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077346.793004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077346.793006:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077346.793013:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4332 00000100:00100000:0.0:1769077346.793017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077346/real 1769077346] req@ffff995b8455a680 x1855007616719232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077346.793017:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077346.793027:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077346.793038:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077346.793061:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616720000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077346.793072:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616719232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077346.793094:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077346.793102:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077346.793105:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616719232:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077346.793134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616719360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077346.793157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077346.793168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077346/real 1769077346] req@ffff995b8455b100 x1855007616719360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077346.793178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616719360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077346.793185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077346.793187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077346.793190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616719360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077346.793195:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616719488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077346.793207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077346.793214:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077346/real 1769077346] req@ffff995b8455aa00 x1855007616719488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077346.793222:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616719488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077346.793230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077346.793232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077346.793234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616719488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077346.793238:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616719616:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077346.793251:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077346.793261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077346/real 1769077346] req@ffff995b84558000 x1855007616719616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077346.793274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616719616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077346.793285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077346.793288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077346.793292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616719616:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077346.793298:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616719744:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077346.793319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077346.793345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077346/real 1769077346] req@ffff995b84559880 x1855007616719744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077346.793359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616719744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077346.793369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077346.793373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077346.793376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616719744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077346.793383:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616719872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077346.793401:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077346.793409:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077346/real 1769077346] req@ffff995b84559c00 x1855007616719872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077346.793422:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616719872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077346.793428:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077346.793430:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077346.793433:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616719872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077346.793437:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616720000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077346.793447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077346.793494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077346/real 1769077346] req@ffff995b8455a300 x1855007616720000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077346.793505:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616720000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077346.793511:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077346.793513:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077346.793515:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616720000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077347.496608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077347.496678:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077347.816506:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077347.816571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077347.816611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077347.816613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077348.520593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077348.520658:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077348.840622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077348.840624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077348.840780:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077348.840815:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077349.544594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077349.544624:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077349.864585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077349.864585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077349.864610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077349.864645:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077350.568574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077350.568638:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077350.888592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077350.888607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077350.888642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077350.888643:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077351.592610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077351.592698:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077351.912630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077351.912631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077351.912634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077351.912680:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077351.912719:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077351.912726:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077351.912731:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077351.912741:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4337 00000100:00080000:1.0:1769077351.912752:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077351.912760:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077351.912809:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616720128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077351.912844:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077351.912848:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077351.912850:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077351.912855:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4337 00000100:00080000:1.0:1769077351.912862:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077351.912870:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077351.912883:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077351.912904:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616720128:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077351.912905:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616720256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077351.912930:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077351.912934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077351.912936:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077351.912941:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4337 00000100:00080000:1.0:1769077351.912943:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077351.912946:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077351.912950:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077351.912973:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616720384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077351.912980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077351.912983:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077351.912984:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077351.912989:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4337 00000100:00080000:1.0:1769077351.912992:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077351.912997:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077351.912999:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077351.913007:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077351.913020:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616720512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077351.913028:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077351.913031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077351.913032:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077351.913039:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4337 00000100:00080000:1.0:1769077351.913044:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077351.913049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077351/real 1769077351] req@ffff995b868c7480 x1855007616720128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077351.913054:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077351.913071:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077351.913078:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616720128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077351.913089:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616720640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077351.913089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077351.913101:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077351.913109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616720128:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077351.913133:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077351.913136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077351.913138:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077351.913143:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4337 00000100:00080000:1.0:1769077351.913148:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077351.913150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616720256:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077351.913164:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077351.913172:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077351.913176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077351.913193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077351/real 1769077351] req@ffff995b868c4700 x1855007616720256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077351.913200:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616720768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077351.913208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616720256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077351.913240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077351.913246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077351.913248:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077351.913252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616720256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077351.913259:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616720384:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077351.913267:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077351.913271:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769077351.913274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077351.913279:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4337 00000100:00080000:1.0:1769077351.913288:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077351.913296:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077351.913299:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077351/real 1769077351] req@ffff995b868c5500 x1855007616720384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077351.913304:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077351.913316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616720384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077351.913326:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616720896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077351.913329:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077351.913349:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077351.913357:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616720384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077351.913371:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616720512:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077351.913429:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077351.913443:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077351/real 1769077351] req@ffff995b868c4a80 x1855007616720512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077351.913484:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616720512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077351.913495:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077351.913501:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077351.913504:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616720512:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077351.913512:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616720640:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077351.913527:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077351.913535:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077351/real 1769077351] req@ffff995b868c6d80 x1855007616720640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077351.913543:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616720640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077351.913549:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077351.913552:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077351.913553:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616720640:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077351.913568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616720768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077351.913578:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077351.913584:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077351/real 1769077351] req@ffff995b868c7800 x1855007616720768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077351.913592:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616720768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077351.913598:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077351.913601:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077351.913603:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616720768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077351.913606:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616720896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077351.913616:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077351.913621:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077351/real 1769077351] req@ffff995b868c6300 x1855007616720896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077351.913629:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616720896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077351.913635:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077351.913638:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077351.913640:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616720896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077352.616583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077352.616644:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077352.936616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077352.936616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077352.936626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077352.936670:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077353.640597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077353.640668:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077353.960545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077353.960562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077353.960582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077353.960599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077354.664548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077354.664659:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077354.984586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077354.984593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077354.984612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077354.984670:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077355.688595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077355.688654:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077356.008540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077356.008562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077356.008612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077356.008614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077356.712571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077356.712581:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077357.032593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077357.032627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077357.032628:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077357.032629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077357.032637:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077357.032643:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077357.032654:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4342 00000100:00100000:1.0:1769077357.032663:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077357.032666:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077357.032674:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077357.032714:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616721024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077357.032738:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077357.032741:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077357.032743:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077357.032746:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4342 00000100:00080000:0.0:1769077357.032749:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077357.032752:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077357.032760:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077357.032769:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616721152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077357.032781:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616721024:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077357.032783:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077357.032799:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077357.032801:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077357.032804:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4342 00000100:00080000:0.0:1769077357.032806:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077357.032808:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077357.032814:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077357.032827:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616721280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077357.032834:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077357.032836:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077357.032839:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077357.032841:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4342 00000100:00080000:0.0:1769077357.032843:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077357.032846:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077357.032849:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077357.032856:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616721408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077357.032861:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077357.032863:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077357.032865:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077357.032868:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4342 00000100:00080000:0.0:1769077357.032871:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077357.032873:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077357.032876:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077357.032886:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077357.032887:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616721536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077357.032899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077357.032901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077357.032904:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077357.032908:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4342 00000100:00080000:0.0:1769077357.032911:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077357.032916:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077357.032919:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077357.032938:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616721664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077357.032943:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077357.032946:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077357.032947:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077357.032950:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4342 00000100:00080000:0.0:1769077357.032960:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077357.032968:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077357/real 1769077357] req@ffff995b84641500 x1855007616721024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077357.032973:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077357.032994:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077357.033005:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616721024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077357.033014:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616721792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077357.033020:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077357.033030:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077357.033039:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616721024:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077357.033082:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616721152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077357.033121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077357.033138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077357/real 1769077357] req@ffff995b84641180 x1855007616721152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077357.033157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616721152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077357.033165:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077357.033167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077357.033170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616721152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077357.033175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616721280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077357.033192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077357.033200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077357/real 1769077357] req@ffff995b84643800 x1855007616721280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077357.033208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616721280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077357.033216:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077357.033220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077357.033223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616721280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077357.033231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616721408:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077357.033252:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077357.033264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077357/real 1769077357] req@ffff995b84643480 x1855007616721408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077357.033279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616721408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077357.033312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077357.033317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077357.033321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616721408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077357.033328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616721536:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077357.033352:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077357.033365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077357/real 1769077357] req@ffff995b84642d80 x1855007616721536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077357.033378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616721536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077357.033389:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077357.033393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077357.033396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616721536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077357.033402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616721664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077357.033430:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077357.033439:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077357/real 1769077357] req@ffff995b84643100 x1855007616721664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077357.033448:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616721664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077357.033477:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077357.033479:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077357.033500:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616721664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077357.033505:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616721792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077357.033521:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077357.033528:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077357/real 1769077357] req@ffff995b84643b80 x1855007616721792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077357.033540:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616721792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077357.033549:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077357.033553:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077357.033556:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616721792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077357.736563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077357.736603:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077358.056530:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077358.056541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077358.056542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077358.056570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077358.760541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077358.760609:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077359.080592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077359.080593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077359.080625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077359.080655:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077359.784561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077359.784610:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077360.104528:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077360.104558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077360.104582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077360.104589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077360.808560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077360.808668:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077361.128593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077361.128596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077361.128607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077361.128692:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077361.832588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077361.832662:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077362.152523:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077362.152565:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077362.152572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077362.152575:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077362.152582:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077362.152596:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4347 00000100:00100000:2.0:1769077362.152599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077362.152600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077362.152607:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077362.152620:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077362.152667:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616721920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077362.152700:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077362.152703:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077362.152705:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077362.152709:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4347 00000100:00080000:1.0:1769077362.152714:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077362.152719:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077362.152729:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077362.152737:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616721920:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077362.152745:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616722048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077362.152754:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077362.152756:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077362.152758:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077362.152762:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4347 00000100:00080000:1.0:1769077362.152767:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077362.152769:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077362.152773:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077362.152789:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616722176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077362.152797:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077362.152799:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077362.152801:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077362.152803:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4347 00000100:00080000:1.0:1769077362.152812:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077362.152814:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077362.152818:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077362.152824:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077362.152834:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616722304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077362.152843:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077362.152846:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077362.152848:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077362.152851:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4347 00000100:00080000:1.0:1769077362.152859:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077362.152862:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077362.152866:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077362.152876:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077362/real 1769077362] req@ffff995b868c7100 x1855007616721920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077362.152877:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616722432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077362.152953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077362.152955:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616721920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077362.152966:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077362.152971:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077362.152986:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4347 00000100:00080000:2.0:1769077362.152988:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077362.152999:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077362.153001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077362.153010:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077362.153011:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616721920:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769077362.153024:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077362.153038:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616722560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077362.153045:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077362.153048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077362.153049:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077362.153054:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4347 00000100:00100000:2.0:1769077362.153058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722048:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077362.153060:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077362.153069:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077362.153073:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077362.153085:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616722688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077362.153096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077362.153113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077362/real 1769077362] req@ffff995b868c6a00 x1855007616722048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077362.153124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616722048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077362.153133:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077362.153136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077362.153138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077362.153143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077362.153154:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077362.153161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077362/real 1769077362] req@ffff995b868c5c00 x1855007616722176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077362.153169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616722176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077362.153181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077362.153184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077362.153186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077362.153189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722304:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077362.153212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077362.153222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077362/real 1769077362] req@ffff995b868c6300 x1855007616722304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077362.153230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616722304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077362.153237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077362.153239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077362.153242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722304:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077362.153246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722432:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077362.153259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077362.153265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077362/real 1769077362] req@ffff995b868c7800 x1855007616722432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077362.153273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616722432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077362.153279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077362.153280:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077362.153283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722432:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077362.153287:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616722560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077362.153298:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077362.153303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077362/real 1769077362] req@ffff995b868c6d80 x1855007616722560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077362.153310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616722560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077362.153316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077362.153318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077362.153320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616722560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077362.153334:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616722688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077362.153345:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077362.153356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077362/real 1769077362] req@ffff995b868c4a80 x1855007616722688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077362.153370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616722688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077362.153378:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077362.153382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077362.153385:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616722688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077362.856552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077362.856630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077363.176568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077363.176568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077363.176569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077363.176601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077363.880558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077363.880653:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077364.200578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077364.200578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077364.200704:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077364.200725:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077364.904521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077364.904614:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077365.224534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077365.224559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077365.224559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077365.224560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077365.928526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077365.928581:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077366.248563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077366.248563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077366.248565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077366.248671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077366.952609:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077366.952687:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077367.272599:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077367.272605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077367.272618:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769077367.272618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077367.272619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077367.272627:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077367.272646:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4352 00000100:00080000:0.0:1769077367.272661:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077367.272670:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077367.272673:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077367.272721:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616722816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077367.272756:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077367.272761:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077367.272763:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077367.272770:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4352 00000100:00080000:0.0:1769077367.272776:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077367.272782:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077367.272794:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616722816:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077367.272795:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077367.272827:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616722944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077367.272842:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077367.272847:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077367.272849:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077367.272855:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4352 00000100:00080000:0.0:1769077367.272860:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077367.272866:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077367.272873:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077367.272878:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077367.272897:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616723072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077367.272918:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077367/real 1769077367] req@ffff995b84641880 x1855007616722816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077367.272922:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077367.272942:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077367.272946:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077367.272951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616722816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077367.272955:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4352 00000100:00080000:0.0:1769077367.272961:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077367.272971:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077367.272987:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769077367.272989:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077367.273000:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077367.273007:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616722816:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077367.273012:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616723200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077367.273026:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077367.273031:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077367.273034:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077367.273040:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4352 00000100:00100000:2.0:1769077367.273045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722944:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077367.273048:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077367.273059:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077367.273067:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077367.273088:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616723328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077367.273099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077367.273131:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077367.273138:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1769077367.273139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077367/real 1769077367] req@ffff995b84640700 x1855007616722944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077367.273144:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077367.273162:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4352 00000100:00100000:2.0:1769077367.273163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616722944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077367.273178:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077367.273185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077367.273187:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077367.273196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077367.273200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616722944:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769077367.273201:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077367.273210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077367.273222:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616723456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077367.273233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077367.273242:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077367.273247:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077367.273251:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077367.273255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077367/real 1769077367] req@ffff995b84642a00 x1855007616723072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077367.273262:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4352 00000100:00080000:0.0:1769077367.273277:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077367.273281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616723072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077367.273290:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077367.273295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769077367.273300:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077367.273302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077367.273309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077367.273317:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723200:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077367.273325:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616723584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077367.273338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077367.273358:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077367/real 1769077367] req@ffff995b84643b80 x1855007616723200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077367.273371:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616723200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077367.273379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077367.273382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077367.273384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723200:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077367.273400:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723328:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077367.273412:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077367.273418:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077367/real 1769077367] req@ffff995b84643100 x1855007616723328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077367.273426:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616723328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077367.273432:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077367.273434:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077367.273436:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723328:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077367.273440:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616723456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077367.273450:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077367.273476:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077367/real 1769077367] req@ffff995b84642d80 x1855007616723456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077367.273484:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616723456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077367.273490:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077367.273492:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077367.273494:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616723456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077367.273497:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616723584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077367.273510:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077367.273516:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077367/real 1769077367] req@ffff995b84643480 x1855007616723584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077367.273529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616723584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077367.273535:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077367.273537:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077367.273538:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616723584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077367.976519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077367.976600:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077368.296550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077368.296550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077368.296551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077368.296579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077369.000573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077369.000637:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077369.320533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077369.320537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077369.320562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077369.320563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077370.024549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077370.024600:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077370.344562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077370.344562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077370.344562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077370.344565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077371.048566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077371.048626:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077371.368535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077371.368536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077371.368555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077371.368566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077372.072526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077372.072592:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077372.392635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077372.392644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077372.392646:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077372.392652:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077372.392715:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077372.392721:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077372.392726:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077372.392740:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4358 00000100:00080000:1.0:1769077372.392753:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077372.392760:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077372.392799:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616723712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077372.392829:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077372.392832:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077372.392834:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077372.392838:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4358 00000100:00080000:1.0:1769077372.392840:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077372.392843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077372.392851:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077372.392863:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616723840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077372.392874:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077372.392877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077372.392879:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077372.392890:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4358 00000100:00100000:2.0:1769077372.392891:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616723712:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077372.392900:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077372.392913:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077372.392918:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077372.392929:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616723968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077372.392937:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077372.392940:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077372.392942:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077372.392944:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4358 00000100:00080000:1.0:1769077372.392949:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077372.392953:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077372.392957:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077372.392967:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616724096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077372.392974:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077372.392977:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077372.392978:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077372.392983:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4358 00000100:00080000:1.0:1769077372.392987:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077372.392993:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077372.392998:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077372.393004:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077372.393015:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616724224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077372.393023:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077372.393025:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077372.393027:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077372.393047:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4358 00000100:00080000:1.0:1769077372.393054:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077372.393059:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077372.393066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077372/real 1769077372] req@ffff995b868c5500 x1855007616723712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077372.393067:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077372.393089:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616724352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077372.393102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616723712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077372.393105:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077372.393118:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077372.393133:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077372.393133:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077372.393158:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4358 00000100:00080000:2.0:1769077372.393160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077372.393175:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077372.393179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616723712:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077372.393187:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077372.393196:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077372.393208:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616724480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077372.393215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077372.393251:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077372.393263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077372/real 1769077372] req@ffff995b868c4700 x1855007616723840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077372.393273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616723840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077372.393280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077372.393283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077372.393285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077372.393291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077372.393304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077372.393311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077372/real 1769077372] req@ffff995b868c7480 x1855007616723968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077372.393320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616723968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077372.393328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077372.393330:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077372.393332:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616723968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077372.393337:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724096:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077372.393362:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077372.393370:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077372/real 1769077372] req@ffff995b868c4a80 x1855007616724096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077372.393380:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616724096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077372.393390:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077372.393394:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077372.393399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724096:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077372.393406:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724224:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077372.393430:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077372.393444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077372/real 1769077372] req@ffff995b868c6d80 x1855007616724224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077372.393492:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616724224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077372.393501:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077372.393504:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077372.393506:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724224:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077372.393512:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616724352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077372.393531:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077372.393540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077372/real 1769077372] req@ffff995b868c7800 x1855007616724352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077372.393549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616724352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077372.393561:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077372.393564:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077372.393567:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616724352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077372.393584:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616724480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077372.393596:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077372.393603:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077372/real 1769077372] req@ffff995b868c6300 x1855007616724480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077372.393612:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616724480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077372.393618:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077372.393621:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077372.393623:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616724480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077373.096561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077373.096582:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077373.416599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077373.416601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077373.416602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077373.416664:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077374.120546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077374.120612:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077374.440558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077374.440558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077374.440565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077374.440570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077375.144550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077375.144619:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077375.464537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077375.464537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077375.464548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077375.464565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077376.168500:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077376.168569:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077376.488560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077376.488560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077376.488566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077376.488568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077377.192574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077377.192623:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077377.512535:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077377.512536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077377.512536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077377.512541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077377.512546:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077377.512552:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077377.512560:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4363 00000100:00080000:1.0:1769077377.512571:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077377.512576:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077377.512585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077377.512615:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616724608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077377.512638:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077377.512642:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077377.512643:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077377.512646:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4363 00000100:00080000:1.0:1769077377.512649:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077377.512652:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077377.512662:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077377.512673:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616724608:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077377.512679:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616724736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077377.512694:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077377.512698:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077377.512700:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077377.512703:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4363 00000100:00080000:1.0:1769077377.512706:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077377.512710:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077377.512714:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077377.512724:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616724864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077377.512731:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077377.512734:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077377.512735:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077377.512738:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4363 00000100:00080000:1.0:1769077377.512740:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077377.512745:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077377.512751:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077377.512756:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077377.512765:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616724992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077377.512772:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077377.512774:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077377.512776:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077377.512778:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4363 00000100:00080000:1.0:1769077377.512781:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077377.512784:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077377/real 1769077377] req@ffff995b868c5c00 x1855007616724608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077377.512785:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077377.512792:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077377.512805:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616725120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077377.512809:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616724608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077377.512818:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077377.512824:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077377.512826:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077377.512826:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077377.512834:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4363 00000100:00080000:2.0:1769077377.512835:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077377.512837:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077377.512841:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616724608:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077377.512843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077377.512851:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077377.512859:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616725248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077377.512866:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077377.512871:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769077377.512871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724736:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077377.512874:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077377.512877:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4363 00000100:00080000:1.0:1769077377.512879:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077377.512884:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077377.512889:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077377.512898:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616725376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077377.512905:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077377.512919:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077377/real 1769077377] req@ffff995b868c6a00 x1855007616724736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077377.512928:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616724736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077377.512934:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077377.512937:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077377.512939:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077377.512970:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077377.512991:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077377.512999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077377/real 1769077377] req@ffff995b868c7100 x1855007616724864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077377.513012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616724864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077377.513019:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077377.513022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077377.513024:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077377.513028:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724992:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077377.513039:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077377.513044:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077377/real 1769077377] req@ffff995b868c6300 x1855007616724992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077377.513052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616724992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077377.513064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077377.513066:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077377.513068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616724992:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077377.513072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616725120:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077377.513082:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077377.513088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077377/real 1769077377] req@ffff995b868c7800 x1855007616725120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077377.513095:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616725120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077377.513106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077377.513109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077377.513111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616725120:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077377.513114:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616725248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077377.513123:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077377.513129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077377/real 1769077377] req@ffff995b868c6d80 x1855007616725248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077377.513136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616725248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077377.513141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077377.513143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077377.513145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616725248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077377.513156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616725376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077377.513166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077377.513173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077377/real 1769077377] req@ffff995b868c4a80 x1855007616725376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077377.513186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616725376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077377.513194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077377.513197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077377.513201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616725376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077378.216557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077378.216606:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077378.536577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077378.536577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077378.536578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077378.536612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077379.240550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077379.240615:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077379.560562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077379.560562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077379.560575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077379.560616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077380.264540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077380.264585:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077380.584556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077380.584570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077380.584579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077380.584607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077381.288585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077381.288654:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077381.608540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077381.608540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077381.608541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077381.608573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077382.312562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077382.312627:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077382.632564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077382.632564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077382.632571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077382.632594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077382.632611:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077382.632617:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077382.632622:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077382.632630:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4368 00000100:00080000:3.0:1769077382.632641:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077382.632649:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077382.632691:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616725504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077382.632720:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077382.632724:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077382.632725:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077382.632728:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4368 00000100:00080000:3.0:1769077382.632732:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077382.632736:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077382.632744:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077382.632756:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616725632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077382.632766:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616725504:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077382.632766:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077382.632771:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077382.632776:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077382.632779:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4368 00000100:00080000:3.0:1769077382.632781:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077382.632783:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077382.632786:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077382.632796:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616725760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077382.632802:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077382.632804:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077382.632805:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077382.632808:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4368 00000100:00080000:3.0:1769077382.632821:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077382.632824:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077382.632827:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077382.632835:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077382.632837:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616725888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077382.632849:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077382.632854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077382.632867:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077382.632872:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4368 00000100:00080000:3.0:1769077382.632877:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077382.632885:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077382/real 1769077382] req@ffff995b84558700 x1855007616725504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077382.632888:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077382.632899:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077382.632907:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616725504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077382.632915:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616726016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077382.632916:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077382.632925:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077382.632929:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077382.632932:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616725504:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077382.632933:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077382.632937:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077382.632942:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4368 00000100:00080000:3.0:1769077382.632945:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077382.632949:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077382.632952:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077382.632960:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616725632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077382.632963:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616726144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077382.632972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077382.632975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077382.632977:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077382.632980:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4368 00000800:00000100:2.0:1769077382.632984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077382.632985:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077382.632990:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077382.632994:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077382.632999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077382/real 1769077382] req@ffff995b8455b480 x1855007616725632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077382.633004:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616726272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077382.633037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616725632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077382.633043:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077382.633046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077382.633049:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616725632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077382.633054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616725760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077382.633065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077382.633071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077382/real 1769077382] req@ffff995b8455ad80 x1855007616725760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077382.633079:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616725760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077382.633086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077382.633088:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077382.633089:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616725760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077382.633093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616725888:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077382.633104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077382.633109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077382/real 1769077382] req@ffff995b8455a300 x1855007616725888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077382.633116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616725888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077382.633126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077382.633128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077382.633130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616725888:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077382.633134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726016:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077382.633142:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077382.633148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077382/real 1769077382] req@ffff995b84559c00 x1855007616726016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077382.633155:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616726016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077382.633161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077382.633163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077382.633165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726016:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077382.633175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616726144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077382.633185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077382.633191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077382/real 1769077382] req@ffff995b84559880 x1855007616726144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077382.633199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616726144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077382.633204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077382.633205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077382.633208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616726144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077382.633212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616726272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077382.633221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077382.633235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077382/real 1769077382] req@ffff995b84558000 x1855007616726272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077382.633243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616726272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077382.633248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077382.633250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077382.633252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616726272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077383.336595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077383.336694:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077383.656597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077383.656598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077383.656653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077383.656775:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077384.360576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077384.360651:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077384.680579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077384.680579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077384.680593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077384.680612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077385.384566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077385.384655:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077385.704590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077385.704590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077385.704636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077385.704636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077386.408585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077386.408643:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077386.728543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077386.728543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077386.728543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077386.728608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077387.432592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077387.432645:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077387.752593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077387.752609:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077387.752620:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769077387.752624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077387.752625:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077387.752643:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4373 00000100:00080000:3.0:1769077387.752655:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077387.752664:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077387.752696:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077387.752708:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616726400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077387.752720:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077387.752736:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077387.752740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077387.752742:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077387.752749:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4373 00000100:00080000:3.0:1769077387.752753:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077387.752759:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616726400:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077387.752764:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077387.752804:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077387.752822:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616726528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077387.752833:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077387.752837:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077387.752839:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077387.752842:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4373 00000100:00080000:3.0:1769077387.752854:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077387.752858:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077387.752865:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077387.752872:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077387.752882:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616726656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077387.752894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077387.752897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077387.752899:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077387.752903:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4373 00000100:00080000:3.0:1769077387.752906:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077387.752923:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077387/real 1769077387] req@ffff995b8455aa00 x1855007616726400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077387.752942:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616726400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077387.752954:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077387.752959:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077387.752962:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616726400:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077387.752985:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077387.753004:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077387.753014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077387/real 1769077387] req@ffff995b8455b100 x1855007616726528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077387.753023:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077387.753036:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077387.753041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616726528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077387.753051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077387.753052:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616726784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077387.753068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077387.753074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077387.753079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077387.753097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077387.753107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077387/real 1769077387] req@ffff995b8455a680 x1855007616726656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077387.753115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616726656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077387.753128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077387.753131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077387.753134:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726656:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077387.753188:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077387.753191:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077387.753193:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077387.753197:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4373 00000100:00080000:3.0:1769077387.753201:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077387.753205:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077387.753232:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726784:192.168.204.132@tcp:8:kworker.0 00000008:00100000:3.0:1769077387.753259:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077387.753268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077387.753280:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616726912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077387.753295:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077387.753300:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1769077387.753301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077387/real 1769077387] req@ffff995b84558a80 x1855007616726784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077387.753303:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077387.753317:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4373 00000100:00080000:3.0:1769077387.753326:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077387.753335:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616726784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077387.753336:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077387.753355:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077387.753356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077387.753361:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077387.753370:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726784:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077387.753383:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616727040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077387.753388:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726912:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077387.753410:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077387.753415:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1769077387.753417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077387.753423:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077387.753433:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077387/real 1769077387] req@ffff995b8455bb80 x1855007616726912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077387.753434:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4373 00000100:00080000:3.0:1769077387.753451:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077387.753485:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077387.753488:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616726912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077387.753494:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077387.753500:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077387.753505:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077387.753513:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616727168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077387.753518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616726912:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077387.753552:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616727040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077387.753570:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077387.753583:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077387/real 1769077387] req@ffff995b84559500 x1855007616727040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077387.753596:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616727040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077387.753605:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077387.753609:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077387.753613:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616727040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077387.753620:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616727168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077387.753641:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077387.753650:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077387/real 1769077387] req@ffff995b84558a80 x1855007616727168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077387.753659:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616727168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077387.753664:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077387.753667:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077387.753669:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616727168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077388.456593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077388.456664:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077388.776592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077388.776626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077388.776654:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077388.776677:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077389.480554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077389.480621:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077389.800619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077389.800619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077389.800619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077389.800657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077390.504551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077390.504679:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077390.824537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077390.824545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077390.824575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077390.824596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077391.528571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077391.528662:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077391.848546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077391.848557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077391.848575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077391.848598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077392.552563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077392.552608:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077392.872522:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077392.872551:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077392.872558:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769077392.872563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077392.872563:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077392.872575:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4378 00000100:00100000:0.0:1769077392.872581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077392.872587:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077392.872595:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077392.872602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077392.872640:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616727296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077392.872660:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077392.872663:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077392.872664:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077392.872666:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4378 00000100:00080000:3.0:1769077392.872669:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077392.872672:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077392.872680:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077392.872717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616727296:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077392.872718:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616727424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077392.872736:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077392.872740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077392.872742:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077392.872747:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4378 00000100:00080000:3.0:1769077392.872752:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077392.872757:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077392.872764:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077392.872780:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616727552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077392.872789:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077392.872794:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077392.872794:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077392.872799:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077392.872804:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4378 00000100:00080000:3.0:1769077392.872807:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077392.872811:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077392.872817:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077392.872828:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616727680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077392.872835:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077392.872840:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077392/real 1769077392] req@ffff995b8455a680 x1855007616727296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077392.872840:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077392.872844:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077392.872851:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4378 00000100:00080000:3.0:1769077392.872856:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077392.872861:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077392.872864:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077392.872869:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616727296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077392.872875:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616727808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077392.872887:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077392.872888:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077392.872895:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077392.872897:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077392.872900:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077392.872904:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4378 00000100:00080000:3.0:1769077392.872908:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077392.872912:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616727296:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077392.872915:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077392.872921:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077392.872934:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616727936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077392.872945:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077392.872946:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616727424:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077392.872953:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077392.872957:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077392.872963:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4378 00000100:00080000:3.0:1769077392.872969:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077392.872973:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077392.872980:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077392.872982:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077392.872995:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616728064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077392.873002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077392/real 1769077392] req@ffff995b8455b100 x1855007616727424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077392.873020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616727424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077392.873031:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077392.873035:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077392.873040:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616727424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077392.873046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616727552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077392.873063:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077392.873072:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077392/real 1769077392] req@ffff995b8455aa00 x1855007616727552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077392.873084:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616727552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077392.873095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077392.873098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077392.873102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616727552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077392.873108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616727680:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077392.873125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077392.873136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077392/real 1769077392] req@ffff995b84558000 x1855007616727680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077392.873149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616727680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077392.873159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077392.873162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077392.873165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616727680:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077392.873172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616727808:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077392.873191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077392.873204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077392/real 1769077392] req@ffff995b84559880 x1855007616727808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077392.873216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616727808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077392.873226:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077392.873229:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077392.873232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616727808:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077392.873249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616727936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077392.873265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077392.873275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077392/real 1769077392] req@ffff995b84559c00 x1855007616727936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077392.873287:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616727936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077392.873295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077392.873299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077392.873302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616727936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077392.873309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616728064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077392.873322:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077392.873331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077392/real 1769077392] req@ffff995b8455a300 x1855007616728064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077392.873342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616728064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077392.873351:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077392.873355:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077392.873358:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616728064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077393.576548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077393.576619:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077393.896554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077393.896576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077393.896599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077393.896611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077394.600556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077394.600616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077394.920588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077394.920735:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077394.922225:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077394.922242:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077395.624548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077395.624609:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077395.944566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077395.944579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077395.944595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077395.944613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077396.648581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077396.648671:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077396.968550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077396.968552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077396.968559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077396.968575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077397.672569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077397.672671:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077397.992561:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077397.992568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077397.992569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077397.992570:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077397.992586:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077397.992596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077397.992619:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4383 00000100:00100000:2.0:1769077397.992631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077397.992632:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077397.992642:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077397.992680:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616728192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077397.992706:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077397.992710:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077397.992711:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077397.992715:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4383 00000100:00080000:1.0:1769077397.992718:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077397.992722:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077397.992730:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077397.992732:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616728192:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077397.992741:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616728320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077397.992750:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077397.992753:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077397.992754:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077397.992757:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4383 00000100:00080000:1.0:1769077397.992759:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077397.992761:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077397.992764:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077397.992773:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616728448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077397.992779:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077397.992781:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077397.992783:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077397.992786:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4383 00000100:00080000:1.0:1769077397.992788:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077397.992791:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077397.992795:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077397.992803:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616728576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077397.992809:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077397.992811:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077397.992812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077397.992815:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4383 00000100:00080000:1.0:1769077397.992819:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077397.992821:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077397.992824:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077397.992832:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616728704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077397.992839:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077397.992841:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077397.992842:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077397.992845:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4383 00000100:00080000:1.0:1769077397.992849:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077397.992853:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077397.992859:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077397.992865:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616728320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077397.992876:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616728832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077397.992889:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077397.992894:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077397.992897:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077397.992901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616728448:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077397.992906:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4383 00000100:00080000:1.0:1769077397.992914:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077397.992917:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077397.992925:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077397.992930:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616728576:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077397.992944:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616728960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077397.992951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616728704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077397.992987:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616728832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077397.993013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616728960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077398.696564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077398.696679:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077399.016579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077399.016612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077399.016625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077399.016669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077399.720550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077399.720646:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077400.040605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077400.040609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077400.040632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077400.040662:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077400.744569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077400.744640:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077401.064622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077401.064622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077401.064624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077401.064773:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:1.0:1769077401.064870:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769077401.064911:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077401.064929:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077401.064990:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077401.065000:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077401.065008:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077401.065016:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077401.065024:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077401.065031:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769077401.065121:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077397/real 1769077401] req@ffff995b868c7800 x1855007616728832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077401.065157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616728832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077401.065170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077401.065177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077401.065181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616728832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077401.065190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077397/real 1769077401] req@ffff995b868c6300 x1855007616728960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077401.065204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616728960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077401.065214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077401.065218:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077401.065222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616728960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077401.065230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077397/real 1769077401] req@ffff995b868c4700 x1855007616728320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077401.065244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616728320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077401.065261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077401.065265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077401.065270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616728320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077401.065277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077397/real 1769077401] req@ffff995b868c5500 x1855007616728448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077401.065290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616728448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077401.065301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077401.065305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077401.065308:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616728448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077401.065315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077397/real 1769077401] req@ffff995b868c4a80 x1855007616728576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077401.065327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616728576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077401.065336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077401.065339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077401.065342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616728576:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077401.065349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077397/real 1769077401] req@ffff995b868c6d80 x1855007616728704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077401.065360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616728704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077401.065369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077401.065373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077401.065376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616728704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077401.065382:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077397/real 1769077401] req@ffff995b868c7480 x1855007616728192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077401.065395:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616728192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077401.065402:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077401.065405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077401.065409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616728192:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077401.768569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077401.768627:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077402.088563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077402.088575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077402.088619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077402.088644:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077402.792550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077402.792618:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077403.112575:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077403.112575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077403.112586:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769077403.112587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077403.112590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077403.112600:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4388 00000100:00100000:1.0:1769077403.112606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077403.112613:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077403.112623:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077403.112630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077403.112666:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616729088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077403.112697:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077403.112702:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077403.112705:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077403.112710:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4388 00000100:00080000:0.0:1769077403.112715:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077403.112721:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077403.112735:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077403.112739:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616729088:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077403.112768:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616729216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077403.112782:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077403.112786:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077403.112788:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077403.112792:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4388 00000100:00080000:0.0:1769077403.112798:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077403.112812:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769077403.112818:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077403.112826:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077403.112843:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616729344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077403.112859:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077403.112863:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077403.112865:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077403.112871:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4388 00000100:00100000:3.0:1769077403.112871:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077403/real 1769077403] req@ffff995b84643800 x1855007616729088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077403.112879:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077403.112886:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077403.112894:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077403.112906:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616729088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077403.112908:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616729472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077403.112936:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077403.112937:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077403.112944:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077403.112948:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077403.112949:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077403.112954:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4388 00000100:00080000:0.0:1769077403.112958:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077403.112966:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077403.112969:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616729088:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077403.112975:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077403.112990:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616729600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077403.113002:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077403.113003:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616729216:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077403.113010:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077403.113014:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077403.113020:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4388 00000100:00080000:0.0:1769077403.113026:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077403.113031:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077403.113038:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077403.113038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077403.113056:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616729728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077403.113060:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077403/real 1769077403] req@ffff995b84641180 x1855007616729216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077403.113075:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077403.113081:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077403.113084:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077403.113085:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616729216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077403.113092:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4388 00000100:00080000:0.0:1769077403.113097:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077403.113119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077403.113122:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077403.113128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769077403.113131:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077403.113135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616729216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077403.113146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616729344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077403.113148:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616729856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077403.113177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077403.113192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077403/real 1769077403] req@ffff995b84641500 x1855007616729344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077403.113211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616729344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077403.113223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077403.113226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077403.113230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616729344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077403.113237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616729472:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077403.113259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077403.113270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077403/real 1769077403] req@ffff995b84642680 x1855007616729472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077403.113282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616729472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077403.114492:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077403.114501:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077403.114506:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616729472:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077403.114549:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616729600:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077403.114604:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077403.114621:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077403/real 1769077403] req@ffff995b84640e00 x1855007616729600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077403.114631:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616729600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077403.114643:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077403.114645:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077403.114648:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616729600:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077403.114653:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616729728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077403.114666:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077403.114673:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077403/real 1769077403] req@ffff995b84641c00 x1855007616729728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077403.114680:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616729728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077403.114686:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077403.114688:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077403.114690:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616729728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077403.114694:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616729856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077403.114705:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077403.114711:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077403/real 1769077403] req@ffff995b84640a80 x1855007616729856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077403.114723:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616729856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077403.114732:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077403.114735:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077403.114739:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616729856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077403.816550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077403.816635:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077404.136579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077404.136579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077404.136610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077404.136649:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077404.840554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077404.840628:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077405.160591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077405.160594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077405.160594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077405.160600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077405.864590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077405.864650:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077406.184564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077406.184564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077406.184570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077406.184584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077406.888555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077406.888676:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077407.208568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077407.208572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077407.208587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077407.208612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077407.912594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077407.912690:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077408.232553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077408.232560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077408.232587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077408.232600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077408.232607:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077408.232615:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077408.232620:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077408.232630:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4393 00000100:00080000:3.0:1769077408.232643:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077408.232651:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077408.232704:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616729984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077408.232733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077408.232737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077408.232739:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077408.232742:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4393 00000100:00080000:3.0:1769077408.232746:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077408.232749:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077408.232759:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077408.232772:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616730112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077408.232782:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077408.232785:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077408.232788:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077408.232792:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4393 00000100:00080000:3.0:1769077408.232796:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077408.232799:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077408.232803:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077408.232811:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616729984:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077408.232822:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616730240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077408.232838:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077408.232843:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077408.232846:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077408.232852:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4393 00000100:00080000:3.0:1769077408.232858:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077408.232865:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077408.232873:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077408.232892:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616730368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077408.232900:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077408.232922:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077408.232926:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077408.232928:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077408.232933:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4393 00000100:00080000:3.0:1769077408.232939:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077408.232943:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077408.232949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077408/real 1769077408] req@ffff995b8455ad80 x1855007616729984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077408.232952:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077408.232974:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616730496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077408.232979:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616729984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077408.232998:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077408.233004:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077408.233005:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077408.233006:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077408.233020:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077408.233020:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4393 00000100:00080000:3.0:1769077408.233031:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077408.233034:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616729984:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077408.233039:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077408.233048:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077408.233050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616730112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077408.233062:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616730624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077408.233073:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077408.233080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077408.233082:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077408.233088:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077408.233098:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4393 00000100:00100000:2.0:1769077408.233099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077408/real 1769077408] req@ffff995b8455b480 x1855007616730112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077408.233108:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077408.233117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616730112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077408.233121:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077408.233135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769077408.233137:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077408.233140:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077408.233147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616730112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077408.233171:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616730752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077408.233176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616730240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077408.233207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077408.233222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077408/real 1769077408] req@ffff995b84558700 x1855007616730240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077408.233233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616730240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077408.233249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077408.233253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077408.233256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616730240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077408.233261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616730368:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077408.233275:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077408.233282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077408/real 1769077408] req@ffff995b8455a300 x1855007616730368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077408.233292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616730368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077408.233300:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077408.233302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077408.233305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616730368:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077408.233309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616730496:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077408.233321:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077408.233330:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077408/real 1769077408] req@ffff995b84559c00 x1855007616730496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077408.233339:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616730496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077408.233347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077408.233349:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077408.233352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616730496:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077408.233357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616730624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077408.233369:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077408.233377:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077408/real 1769077408] req@ffff995b84559880 x1855007616730624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077408.233386:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616730624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077408.233393:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077408.233395:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077408.233397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616730624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077408.233409:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616730752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077408.233422:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077408.233491:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077408/real 1769077408] req@ffff995b84558000 x1855007616730752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077408.233503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616730752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077408.233510:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077408.233513:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077408.233515:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616730752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077408.936542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077408.936581:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077409.256574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077409.256574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077409.256582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077409.256657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077409.960567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077409.960611:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077410.280543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077410.280545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077410.280567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077410.281503:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077410.984544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077410.984592:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077411.304510:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077411.304532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077411.304541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077411.304561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077412.008580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077412.008677:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077412.328510:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077412.328527:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077412.329506:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077412.329510:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077413.032545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077413.032601:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077413.352521:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077413.352529:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077413.352553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077413.352578:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077413.352587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077413.352588:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077413.352594:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077413.352608:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4399 00000100:00080000:2.0:1769077413.352621:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077413.352631:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077413.352666:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616730880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077413.352691:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077413.352695:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077413.352696:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077413.352699:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4399 00000100:00080000:2.0:1769077413.352703:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077413.352707:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077413.352715:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077413.352719:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616730880:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077413.352732:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616731008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077413.352744:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077413.352748:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077413.352751:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077413.352755:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4399 00000100:00080000:2.0:1769077413.352760:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077413.352764:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077413.352771:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077413.352784:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616731136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077413.352793:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077413.352801:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077413.352804:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077413.352807:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077413.352811:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4399 00000100:00080000:2.0:1769077413.352815:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077413.352819:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077413.352823:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077413.352833:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616731264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077413.352840:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077413/real 1769077413] req@ffff995b85acbb80 x1855007616730880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077413.352843:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077413.352854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077413.352858:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077413.352864:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4399 00000100:00100000:3.0:1769077413.352866:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616730880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077413.352869:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077413.352880:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077413.352884:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769077413.352886:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769077413.352893:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077413.352900:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616730880:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077413.352901:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616731392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077413.352914:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077413.352917:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077413.352919:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077413.352922:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4399 00000100:00080000:2.0:1769077413.352925:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077413.352928:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077413.352932:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077413.352937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077413.352941:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616731520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077413.352951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077413.352954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077413.352955:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077413.352958:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4399 00000100:00080000:2.0:1769077413.352964:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077413.352967:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769077413.352971:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077413.352973:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077413.352983:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616731648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077413.352988:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077413/real 1769077413] req@ffff995b85ac8380 x1855007616731008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077413.353003:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616731008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077413.353011:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077413.353014:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077413.353018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077413.353023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077413.353038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077413.353046:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077413/real 1769077413] req@ffff995b85ac9880 x1855007616731136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077413.353054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616731136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077413.353062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077413.353064:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077413.353066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077413.353070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731264:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077413.353082:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077413.353091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077413/real 1769077413] req@ffff995b85ac9c00 x1855007616731264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077413.353104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616731264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077413.353114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077413.353117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077413.353121:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077413.353127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731392:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077413.353148:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077413.353157:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077413/real 1769077413] req@ffff995b85acb800 x1855007616731392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077413.353170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616731392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077413.353176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077413.353179:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077413.353181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077413.353194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616731520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077413.353216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077413.353224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077413/real 1769077413] req@ffff995b85aca300 x1855007616731520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077413.353232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616731520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077413.353237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077413.353240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077413.353242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616731520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077413.353245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616731648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077413.353255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077413.353261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077413/real 1769077413] req@ffff995b85acaa00 x1855007616731648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077413.353268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616731648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077413.353278:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077413.353281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077413.353283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616731648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077414.056610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077414.056676:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077414.376554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077414.376605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077414.376607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077414.376666:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077415.080562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077415.080648:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077415.400526:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077415.400562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077415.400591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077415.401505:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077416.104547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077416.104626:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077416.424540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077416.424541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077416.424567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077416.424591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077417.128531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077417.128611:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077417.448564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077417.448564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077417.448569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077417.448667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077418.152583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077418.152657:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077418.472546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077418.472546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077418.472565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077418.472582:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077418.472588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077418.472588:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077418.472594:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077418.472604:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4404 00000100:00080000:3.0:1769077418.472617:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077418.472624:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077418.472661:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616731776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077418.472685:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077418.472689:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077418.472690:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077418.472694:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4404 00000100:00080000:3.0:1769077418.472697:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077418.472700:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077418.472709:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077418.472714:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616731776:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077418.472721:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616731904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077418.472730:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077418.472732:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077418.472733:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077418.472736:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4404 00000100:00080000:3.0:1769077418.472738:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077418.472740:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077418.472743:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077418.472753:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616732032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077418.472758:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077418.472761:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077418.472762:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077418.472764:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4404 00000100:00080000:3.0:1769077418.472773:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077418.472777:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077418.472780:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077418.472787:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077418.472791:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616732160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077418.472799:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077418.472802:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077418.472803:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077418.472807:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4404 00000100:00080000:3.0:1769077418.472810:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077418.472813:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077418.472816:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077418.472823:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616732288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077418.472828:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077418.472831:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077418.472832:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077418.472836:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077418/real 1769077418] req@ffff995b8455aa00 x1855007616731776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077418.472839:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4404 00000100:00080000:3.0:1769077418.472844:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077418.472850:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077418.472856:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077418.472864:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616731776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077418.472865:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616732416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077418.472891:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077418.472892:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077418.472895:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077418.472900:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077418.472900:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077418.472907:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616731776:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077418.472915:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4404 00000100:00080000:3.0:1769077418.472919:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077418.472922:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077418.472925:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077418.472933:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077418.472936:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616732544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077418.472962:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077418.472974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077418/real 1769077418] req@ffff995b8455b100 x1855007616731904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077418.472983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616731904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077418.472989:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077418.472992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077418.472997:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616731904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077418.473002:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077418.473014:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077418.473020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077418/real 1769077418] req@ffff995b8455a680 x1855007616732032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077418.473031:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616732032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077418.473042:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077418.473046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077418.473050:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077418.473056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077418.473078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077418.473089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077418/real 1769077418] req@ffff995b84558a80 x1855007616732160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077418.473102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616732160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077418.473110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077418.473112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077418.473115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077418.473119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732288:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077418.473133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077418.473157:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077418/real 1769077418] req@ffff995b84559500 x1855007616732288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077418.473167:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616732288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077418.473174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077418.473176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077418.473178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077418.473182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616732416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077418.473193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077418.473200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077418/real 1769077418] req@ffff995b8455bb80 x1855007616732416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077418.473214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616732416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077418.473223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077418.473226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077418.473246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616732416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077418.473293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616732544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077418.473311:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077418.473319:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077418/real 1769077418] req@ffff995b8455aa00 x1855007616732544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077418.473327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616732544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077418.473338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077418.473340:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077418.473342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616732544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077419.176541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077419.176616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077419.496641:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077419.496641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077419.496641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077419.496663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077420.200562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077420.200639:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077420.520628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077420.520630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077420.520630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077420.520652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077421.224540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077421.224597:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077421.544590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077421.544616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077421.544635:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077421.544780:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077422.248553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077422.248643:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077422.568619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077422.568628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077422.568672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077422.568672:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077423.272516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077423.272636:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077423.592547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077423.592553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077423.592557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077423.592580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077423.592601:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077423.592607:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077423.592611:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077423.592619:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4409 00000100:00080000:1.0:1769077423.592629:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077423.592637:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077423.592680:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616732672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077423.592702:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077423.592705:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077423.592706:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077423.592710:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4409 00000100:00080000:1.0:1769077423.592713:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077423.592716:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077423.592724:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077423.592736:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616732800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077423.592748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077423.592750:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077423.592752:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077423.592754:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4409 00000100:00080000:1.0:1769077423.592756:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077423.592759:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077423.592762:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077423.592770:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616732928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077423.592778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077423.592780:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077423.592782:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077423.592786:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4409 00000100:00080000:1.0:1769077423.592790:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077423.592793:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077423.592798:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077423.592803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616732672:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077423.592809:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616733056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077423.592818:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077423.592821:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077423.592823:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077423.592826:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4409 00000100:00080000:1.0:1769077423.592830:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077423.592833:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077423.592837:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077423.592854:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616733184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077423.592862:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077423.592865:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077423.592867:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077423.592871:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4409 00000100:00080000:1.0:1769077423.592874:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077423.592878:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077423.592884:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769077423.592884:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077423.592896:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616733312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077423.592903:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077423.592906:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077423.592908:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077423.592910:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4409 00000100:00080000:1.0:1769077423.592913:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077423.592915:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077423.592918:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077423.592925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077423/real 1769077423] req@ffff995b868c7100 x1855007616732672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077423.592928:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616733440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077423.592970:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616732672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077423.592985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077423.593009:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077423.593014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616732672:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077423.593023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077423.593044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077423.593055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077423/real 1769077423] req@ffff995b868c6a00 x1855007616732800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077423.593082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616732800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077423.593089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077423.593093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077423.593095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077423.593109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077423.593125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077423.593151:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077423/real 1769077423] req@ffff995b868c5c00 x1855007616732928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077423.593165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616732928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077423.593206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077423.593212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077423.593216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616732928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077423.593245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733056:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077423.593266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077423.593282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077423/real 1769077423] req@ffff995b868c7480 x1855007616733056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077423.593293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616733056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077423.593299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077423.593302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077423.593304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077423.593309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733184:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077423.593318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077423.593324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077423/real 1769077423] req@ffff995b868c6d80 x1855007616733184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077423.593331:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616733184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077423.593338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077423.593339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077423.593341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077423.593345:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616733312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077423.593354:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077423.593360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077423/real 1769077423] req@ffff995b868c4a80 x1855007616733312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077423.593367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616733312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077423.593372:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077423.593374:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077423.593376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616733312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077423.593379:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616733440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077423.593389:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077423.593395:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077423/real 1769077423] req@ffff995b868c5500 x1855007616733440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077423.593403:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616733440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077423.593412:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077423.593415:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077423.593419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616733440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077424.296578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077424.296648:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077424.616596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077424.616607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077424.616607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077424.616660:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077425.320536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077425.320592:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077425.640562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077425.640568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077425.640602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077425.640609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077426.344538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077426.344604:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077426.664566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077426.664566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077426.664566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077426.664566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077427.368557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077427.368608:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077427.688538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077427.688543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077427.688579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077427.688602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077428.392529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077428.392588:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077428.712576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077428.712576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077428.712576:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077428.712576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077428.712588:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077428.712594:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077428.712605:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4414 00000100:00100000:3.0:1769077428.712614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077428.712618:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077428.712628:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077428.712670:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616733568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077428.712704:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077428.712708:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077428.712711:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077428.712716:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4414 00000100:00080000:2.0:1769077428.712720:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077428.712725:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077428.712736:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077428.712738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616733568:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077428.712753:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616733696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077428.712764:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077428.712768:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077428.712770:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077428.712774:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4414 00000100:00080000:2.0:1769077428.712779:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077428.712784:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077428.712789:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077428.712802:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616733824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077428.712811:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077428.712819:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077428.712824:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077428.712826:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077428.712832:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4414 00000100:00080000:2.0:1769077428.712837:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077428.712842:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077428.712849:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077428.712850:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077428/real 1769077428] req@ffff995b85ac8a80 x1855007616733568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077428.712871:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616733568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077428.712872:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616733952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077428.712885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077428.712893:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077428.712894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077428.712898:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616733568:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077428.712901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077428.712906:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077428.712912:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4414 00000100:00080000:2.0:1769077428.712916:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077428.712921:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077428.712924:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733696:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077428.712928:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077428.712947:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616734080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077428.712950:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077428.712964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077428.712968:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077428/real 1769077428] req@ffff995b85acb100 x1855007616733696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077428.712970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077428.712974:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077428.712984:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4414 00000100:00080000:2.0:1769077428.712990:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077428.712995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616733696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077428.712998:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077428.713007:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077428.713011:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077428.713016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077428.713020:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077428.713025:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616734208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077428.713030:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733824:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077428.713050:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077428.713054:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077428.713058:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077428.713062:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077428.713068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077428/real 1769077428] req@ffff995b85acaa00 x1855007616733824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077428.713069:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4414 00000100:00080000:2.0:1769077428.713077:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077428.713099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616733824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077428.713108:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077428.713113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077428.713115:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077428.713118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733824:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077428.713125:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077428.713126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077428.713149:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616734336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077428.713153:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077428.713164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077428/real 1769077428] req@ffff995b85aca300 x1855007616733952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077428.713189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616733952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077428.713196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077428.713199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077428.713201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616733952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077428.713214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734080:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077428.713225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077428.713232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077428/real 1769077428] req@ffff995b85acb800 x1855007616734080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077428.713240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616734080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077428.713246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077428.713249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077428.713251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077428.713255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616734208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077428.713265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077428.713270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077428/real 1769077428] req@ffff995b85ac9c00 x1855007616734208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077428.713278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616734208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077428.713289:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077428.713290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077428.713292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616734208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077428.713296:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616734336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077428.713304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077428.713310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077428/real 1769077428] req@ffff995b85ac9880 x1855007616734336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077428.713319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616734336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077428.713328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077428.713331:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077428.713334:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616734336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077429.416557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077429.416608:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077429.736537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077429.736611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077429.736630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077429.736643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077430.440541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077430.440623:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077430.760562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077430.760562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077430.760586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077430.760601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077431.464540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077431.464587:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077431.784529:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077431.784536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077431.784570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077431.784576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077432.488519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077432.488572:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077432.808571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077432.808571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077432.808572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077432.808604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077433.512555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077433.512624:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077433.832547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077433.832558:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077433.832562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077433.832565:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077433.832571:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077433.832579:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4419 00000100:00100000:1.0:1769077433.832587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077433.832591:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077433.832598:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077433.832614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077433.832636:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616734464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077433.832655:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077433.832659:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077433.832661:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077433.832664:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4419 00000100:00080000:2.0:1769077433.832667:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077433.832670:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077433.832678:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077433.832684:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616734464:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077433.832690:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616734592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077433.832699:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077433.832701:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077433.832702:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077433.832705:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4419 00000100:00080000:2.0:1769077433.832707:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077433.832710:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077433.832713:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077433.832723:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616734720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077433.832730:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077433.832733:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077433.832734:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077433.832736:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4419 00000100:00080000:2.0:1769077433.832738:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077433.832740:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077433.832745:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077433.832753:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616734848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077433.832761:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077433.832762:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077433.832765:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077433.832767:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077433.832770:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4419 00000100:00080000:2.0:1769077433.832773:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077433.832775:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077433.832778:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077433.832785:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616734976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077433.832791:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077433.832793:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077433.832795:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077433.832797:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4419 00000100:00080000:2.0:1769077433.832799:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077433.832802:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077433.832805:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077433.832815:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077433/real 1769077433] req@ffff995b85ac8380 x1855007616734464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077433.832815:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616735104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077433.832832:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077433.832839:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077433.832841:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077433.832847:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4419 00000100:00080000:2.0:1769077433.832851:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077433.832855:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077433.832861:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077433.832872:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616735232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077433.832882:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616734464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077433.832895:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077433.832901:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077433.832906:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616734464:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077433.832933:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077433.832972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077433.832987:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077433/real 1769077433] req@ffff995b85acbb80 x1855007616734592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077433.833019:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616734592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077433.833029:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077433.833033:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077433.833037:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077433.833044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077433.833067:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077433.833079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077433/real 1769077433] req@ffff995b85ac9880 x1855007616734720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077433.833092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616734720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077433.833111:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077433.833116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077433.833120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077433.833139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077433.833162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077433.833173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077433/real 1769077433] req@ffff995b85ac9c00 x1855007616734848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077433.833185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616734848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077433.833196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077433.833199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077433.833203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077433.833210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077433.833231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077433.833243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077433/real 1769077433] req@ffff995b85acb800 x1855007616734976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077433.833257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616734976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077433.833267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077433.833271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077433.833275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616734976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077433.833281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616735104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077433.833309:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077433.833322:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077433/real 1769077433] req@ffff995b85aca300 x1855007616735104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077433.833334:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616735104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077433.833343:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077433.833347:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077433.833350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616735104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077433.833356:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616735232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077433.833375:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077433.833387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077433/real 1769077433] req@ffff995b85acaa00 x1855007616735232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077433.833400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616735232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077433.833409:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077433.833412:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077433.833416:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616735232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077434.536520:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077434.536613:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077434.856602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077434.856613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077434.856617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077434.856648:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077435.560535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077435.560655:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077435.880548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077435.880586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077435.880629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077435.880652:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077436.584552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077436.584626:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077436.904636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077436.904639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077436.904645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077436.904676:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077437.608585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077437.608653:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077437.928535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077437.928570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077437.928583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077437.928597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077438.632555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077438.632618:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077438.952601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077438.952601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077438.952607:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077438.952617:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077438.952622:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077438.952640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077438.952644:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4424 00000100:00080000:1.0:1769077438.952660:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077438.952662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077438.952670:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077438.952714:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616735360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077438.952751:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077438.952756:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077438.952759:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077438.952764:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4424 00000100:00080000:1.0:1769077438.952769:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077438.952774:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077438.952786:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077438.952794:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616735360:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077438.952804:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616735488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077438.952815:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077438.952819:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077438.952821:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077438.952824:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4424 00000100:00080000:1.0:1769077438.952828:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077438.952831:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077438.952835:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077438.952845:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616735616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077438.952852:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077438.952855:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077438.952856:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077438.952858:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4424 00000100:00080000:1.0:1769077438.952869:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077438.952875:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077438.952878:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769077438.952884:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077438.952899:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616735744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077438.952908:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077438.952924:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077438.952927:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077438.952930:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4424 00000100:00080000:1.0:1769077438.952934:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077438.952938:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077438.952945:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077438.952946:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077438/real 1769077438] req@ffff995b868c4700 x1855007616735360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077438.952960:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616735872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077438.952969:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077438.952970:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616735360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077438.952980:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077438.952984:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077438.953000:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4424 00000100:00080000:1.0:1769077438.953005:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077438.953006:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077438.953020:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077438.953020:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077438.953025:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077438.953032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616735360:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077438.953047:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616736000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077438.953060:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616735488:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077438.953062:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077438.953070:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077438.953074:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077438.953080:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4424 00000100:00080000:1.0:1769077438.953085:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077438.953088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077438.953090:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077438.953098:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077438.953106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077438/real 1769077438] req@ffff995b868c6300 x1855007616735488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077438.953117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616735488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077438.953123:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616736128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077438.953126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077438.953145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077438.953162:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616735488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077438.953175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616735616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077438.953202:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077438.953224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077438/real 1769077438] req@ffff995b868c7800 x1855007616735616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077438.953239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616735616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077438.953251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077438.953255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077438.953259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616735616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077438.953265:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616735744:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077438.953287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077438.953307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077438/real 1769077438] req@ffff995b868c5500 x1855007616735744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077438.953322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616735744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077438.953333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077438.953337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077438.953341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616735744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077438.953348:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616735872:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077438.953369:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077438.953381:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077438/real 1769077438] req@ffff995b868c4a80 x1855007616735872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077438.953414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616735872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077438.953425:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077438.953428:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077438.953432:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616735872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077438.953478:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616736000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077438.953503:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077438.953525:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077438/real 1769077438] req@ffff995b868c6d80 x1855007616736000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077438.953539:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616736000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077438.953560:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077438.953564:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077438.953568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616736000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077438.953575:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616736128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077438.953607:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077438.953619:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077438/real 1769077438] req@ffff995b868c7480 x1855007616736128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077438.953645:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616736128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077438.953661:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077438.953665:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077438.953668:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616736128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077439.656578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077439.656675:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077439.976588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077439.976589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077439.976623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077439.976638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077440.680530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077440.680605:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077441.000546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077441.000554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077441.000581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077441.000610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077441.704611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077441.704650:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077442.024602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077442.024604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077442.024605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077442.024654:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077442.728566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077442.728619:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077443.048582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077443.048582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077443.048582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077443.048592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077443.752565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077443.752640:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077444.072585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077444.072585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077444.072586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077444.072586:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077444.072596:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077444.072602:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077444.072613:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4429 00000100:00080000:2.0:1769077444.072625:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077444.072633:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077444.072640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077444.072694:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616736256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077444.072719:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077444.072723:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077444.072724:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077444.072727:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4429 00000100:00080000:2.0:1769077444.072731:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077444.072734:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077444.072742:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077444.072753:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616736384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077444.072760:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077444.072763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077444.072765:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077444.072767:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4429 00000100:00080000:2.0:1769077444.072769:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077444.072774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616736256:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077444.072775:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077444.072781:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077444.072793:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616736512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077444.072801:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077444.072803:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077444.072804:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077444.072808:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4429 00000100:00080000:2.0:1769077444.072810:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077444.072813:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077444.072817:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077444.072851:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616736640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077444.072859:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077444.072861:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077444.072863:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769077444.072869:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077444.072870:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4429 00000100:00080000:2.0:1769077444.072876:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077444.072880:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077444.072884:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077444.072894:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616736768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077444.072901:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077444.072904:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077444.072905:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077444.072909:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4429 00000100:00080000:2.0:1769077444.072911:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077444.072915:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077444.072920:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077444/real 1769077444] req@ffff995b85acb100 x1855007616736256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077444.072921:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077444.072934:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616736896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077444.072943:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077444.072947:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769077444.072949:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616736256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077444.072949:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077444.072957:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4429 00000100:00080000:2.0:1769077444.072961:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077444.072968:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077444.072976:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077444.072978:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077444.072986:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616737024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077444.072989:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077444.072999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616736256:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077444.073010:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616736384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077444.073035:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077444.073048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077444/real 1769077444] req@ffff995b85ac8a80 x1855007616736384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077444.073071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616736384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077444.073081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077444.073085:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077444.073089:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616736384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077444.073116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616736512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077444.073136:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077444.073148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077444/real 1769077444] req@ffff995b85acaa00 x1855007616736512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077444.073162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616736512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077444.073174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077444.073178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077444.073182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616736512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077444.073188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616736640:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077444.073207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077444.073217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077444/real 1769077444] req@ffff995b85aca300 x1855007616736640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077444.073230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616736640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077444.073248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077444.073252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077444.073256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616736640:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077444.073262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616736768:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077444.073279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077444.073289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077444/real 1769077444] req@ffff995b85acb800 x1855007616736768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077444.073302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616736768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077444.073312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077444.073315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077444.073319:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616736768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077444.073325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616736896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077444.073343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077444.073353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077444/real 1769077444] req@ffff995b85ac9c00 x1855007616736896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077444.073366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616736896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077444.073375:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077444.073378:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077444.073382:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616736896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077444.073389:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616737024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077444.073415:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077444.073428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077444/real 1769077444] req@ffff995b85ac9880 x1855007616737024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077444.073442:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616737024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077444.073451:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077444.073474:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077444.073478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616737024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077444.776556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077444.776617:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077445.096561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077445.096590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077445.096626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077445.096653:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077445.800544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077445.800593:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077446.120539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077446.120574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077446.120576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077446.120581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077446.824571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077446.824639:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077447.144567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077447.144568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077447.144571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077447.144571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077447.848549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077447.848600:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077448.168502:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077448.168545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077448.168562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077448.168569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077448.872541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077448.872607:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077449.192541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077449.192541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077449.192546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077449.192568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077449.192591:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077449.192598:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077449.192603:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077449.192612:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4434 00000100:00080000:1.0:1769077449.192622:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077449.192628:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077449.192671:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616737152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077449.192699:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077449.192704:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077449.192706:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077449.192711:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4434 00000100:00080000:1.0:1769077449.192716:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077449.192722:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077449.192733:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077449.192735:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616737152:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077449.192768:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616737280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077449.192782:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077449.192786:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077449.192788:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077449.192793:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4434 00000100:00080000:1.0:1769077449.192806:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077449.192811:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077449.192818:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769077449.192821:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077449.192838:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616737408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077449.192850:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077449.192854:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077449.192857:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077449.192862:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4434 00000100:00080000:1.0:1769077449.192867:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077449.192875:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077449.192876:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077449/real 1769077449] req@ffff995b868c5c00 x1855007616737152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077449.192882:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077449.192903:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616737536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077449.192906:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616737152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077449.192929:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077449.192932:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077449.192936:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077449.192941:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077449.192942:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077449.192948:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616737152:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077449.192952:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4434 00000100:00080000:1.0:1769077449.192960:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077449.192966:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077449.192973:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077449.192978:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616737280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077449.192989:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616737664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077449.193005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077449.193005:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077449.193011:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077449.193014:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077449.193022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077449/real 1769077449] req@ffff995b868c6a00 x1855007616737280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077449.193022:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4434 00000100:00080000:1.0:1769077449.193029:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077449.193038:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077449.193043:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616737280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077449.193046:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077449.193059:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077449.193067:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077449.193068:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616737792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077449.193072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616737280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077449.193086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616737408:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077449.193091:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077449.193097:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077449.193099:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077449.193105:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4434 00000800:00000100:0.0:1769077449.193108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077449.193113:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077449.193121:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077449.193125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077449/real 1769077449] req@ffff995b868c7100 x1855007616737408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077449.193129:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077449.193145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616737408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077449.193150:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616737920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077449.193164:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077449.193172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077449.193177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616737408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077449.193184:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616737536:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077449.193205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077449.193217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077449/real 1769077449] req@ffff995b868c7480 x1855007616737536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077449.193227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616737536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077449.193236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077449.193238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077449.193242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616737536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077449.193257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616737664:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077449.193276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077449.193287:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077449/real 1769077449] req@ffff995b868c6d80 x1855007616737664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077449.193298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616737664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077449.193308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077449.193311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077449.193314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616737664:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077449.193318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616737792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077449.193333:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077449.193341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077449/real 1769077449] req@ffff995b868c4a80 x1855007616737792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077449.193353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616737792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077449.193359:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077449.193360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077449.193362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616737792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077449.193365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616737920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077449.193375:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077449.193380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077449/real 1769077449] req@ffff995b868c5500 x1855007616737920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077449.193388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616737920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077449.193392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077449.193394:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077449.193396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616737920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077449.896536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077449.896620:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077450.216537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077450.216537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077450.216554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077450.216581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077450.920581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077450.920660:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077451.240524:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077451.240536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077451.240541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077451.240573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077451.944575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077451.944651:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077452.264553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077452.264586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077452.264595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077452.264629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077452.968551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077452.968611:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077453.288547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077453.288563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077453.288577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077453.288598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077453.992529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077453.992578:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077454.312558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077454.312558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077454.312578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077454.312593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077454.312614:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077454.312620:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077454.312624:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077454.312633:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4440 00000100:00080000:3.0:1769077454.312643:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077454.312649:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077454.312701:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616738048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077454.312724:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077454.312728:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077454.312730:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077454.312735:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4440 00000100:00080000:3.0:1769077454.312739:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077454.312744:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077454.312755:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077454.312762:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616738048:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077454.312771:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616738176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077454.312790:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077454.312794:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077454.312796:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077454.312800:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4440 00000100:00080000:3.0:1769077454.312804:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077454.312809:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077454.312813:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077454.312824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616738304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077454.312834:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077454.312835:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077454.312842:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077454.312845:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077454.312851:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4440 00000100:00080000:3.0:1769077454.312855:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077454.312860:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077454.312866:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077454.312876:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077454/real 1769077454] req@ffff995b84558000 x1855007616738048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077454.312882:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616738432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077454.312899:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077454.312901:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616738048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077454.312908:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077454.312913:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077454.312916:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077454.312921:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4440 00000100:00080000:0.0:1769077454.312924:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077454.312930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616738048:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077454.312931:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077454.312939:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077454.312944:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077454.312954:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616738560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077454.312962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616738176:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077454.312963:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077454.312969:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077454.312971:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077454.312975:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4440 00000100:00080000:3.0:1769077454.312979:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077454.312982:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077454.312985:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077454.312990:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077454.312996:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616738688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077454.313004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077454/real 1769077454] req@ffff995b84559880 x1855007616738176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077454.313006:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077454.313014:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077454.313017:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077454.313024:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4440 00000100:00100000:0.0:1769077454.313027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616738176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077454.313028:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077454.313036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077454.313044:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077454.313044:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077454.313052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616738176:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769077454.313053:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077454.313063:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616738304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077454.313068:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616738816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077454.313081:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077454.313092:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077454/real 1769077454] req@ffff995b84559c00 x1855007616738304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077454.313101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616738304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077454.313113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077454.313115:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077454.313118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616738304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077454.313122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616738432:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077454.313132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077454.313138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077454/real 1769077454] req@ffff995b8455a300 x1855007616738432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077454.313145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616738432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077454.313152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077454.313153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077454.313155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616738432:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077454.313165:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616738560:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077454.313175:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077454.313181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077454/real 1769077454] req@ffff995b84558700 x1855007616738560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077454.313189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616738560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077454.313195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077454.313196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077454.313199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616738560:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077454.313203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616738688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077454.313219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077454.313226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077454/real 1769077454] req@ffff995b8455b480 x1855007616738688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077454.313233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616738688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077454.313238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077454.313240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077454.313242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616738688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077454.313246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616738816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077454.313257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077454.313262:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077454/real 1769077454] req@ffff995b8455ad80 x1855007616738816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077454.313270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616738816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077454.313275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077454.313276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077454.313278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616738816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077455.016632:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077455.016660:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077455.336549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077455.336549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077455.336553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077455.336604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077456.040569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077456.040644:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077456.360556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077456.360587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077456.360634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077456.360644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077457.064707:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077457.064781:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077457.384544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077457.384549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077457.384559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077457.384589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077458.088533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077458.088615:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077458.408543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077458.408543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077458.408543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077458.408580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077459.112568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077459.112653:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077459.432554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077459.432554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077459.432554:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077459.432564:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077459.432582:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077459.432591:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4445 00000100:00080000:3.0:1769077459.432602:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077459.432608:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077459.432638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077459.432640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077459.432650:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616738944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077459.432674:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077459.432678:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077459.432681:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077459.432685:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4445 00000100:00080000:3.0:1769077459.432690:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077459.432691:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616738944:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077459.432698:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077459.432711:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077459.432728:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616739072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077459.432738:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077459.432741:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077459.432743:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077459.432746:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4445 00000100:00080000:3.0:1769077459.432749:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077459.432752:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077459.432757:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077459.432769:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616739200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077459.432773:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077459.432783:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077459.432786:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077459.432788:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077459.432791:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4445 00000100:00080000:3.0:1769077459.432794:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077459.432797:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077459.432801:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077459.432812:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616739328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077459.432820:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077459/real 1769077459] req@ffff995b8455ad80 x1855007616738944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077459.432822:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077459.432828:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077459.432831:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077459.432839:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4445 00000100:00080000:3.0:1769077459.432842:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077459.432847:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077459.432855:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077459.432871:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616738944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077459.432882:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077459.432886:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077459.432890:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616738944:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077459.432921:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077459.432925:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616739456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077459.432935:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077459.432938:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077459.432941:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077459.432946:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4445 00000800:00000100:1.0:1769077459.432950:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077459.432954:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077459.432964:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077459/real 1769077459] req@ffff995b8455b480 x1855007616739072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077459.432974:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616739072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077459.432982:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077459.432986:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077459.432990:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077459.432996:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077459.433018:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077459.433031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077459/real 1769077459] req@ffff995b84558700 x1855007616739200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077459.433047:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616739200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077459.433055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077459.433059:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769077459.433063:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769077459.433067:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077459.433068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077459.433078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739328:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077459.433084:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616739584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077459.433100:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077459.433111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077459/real 1769077459] req@ffff995b8455a300 x1855007616739328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077459.433124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616739328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077459.433129:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077459.433138:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077459.433142:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077459.433149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077459.433149:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4445 00000100:00080000:1.0:1769077459.433157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077459.433159:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077459.433161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739328:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077459.433169:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077459.433176:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077459.433179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077459.433195:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616739712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077459.433196:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077459.433206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077459/real 1769077459] req@ffff995b84559c00 x1855007616739456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077459.433218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616739456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077459.433226:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077459.433229:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077459.433232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077459.433237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616739584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077459.433247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077459.433254:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077459/real 1769077459] req@ffff995b84559880 x1855007616739584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077459.433266:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616739584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077459.433275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077459.433278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077459.433282:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616739584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077459.433296:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616739712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077459.433317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077459.433329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077459/real 1769077459] req@ffff995b84558000 x1855007616739712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077459.433338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616739712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077459.433344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077459.433346:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077459.433348:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616739712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077460.136535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077460.136670:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077460.456584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077460.456584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077460.456598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077460.456605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077461.160569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077461.160673:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077461.480634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077461.480634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077461.480634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077461.480653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077462.184548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077462.184633:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077462.504632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077462.504636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077462.504640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077462.504646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077463.208609:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077463.208678:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077463.528641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077463.528642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077463.528642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077463.528642:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077464.232579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077464.232679:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077464.552593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077464.552602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077464.552613:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077464.552622:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769077464.552626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077464.552627:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077464.552632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077464.552647:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4450 00000100:00080000:1.0:1769077464.552661:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077464.552672:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077464.552712:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616739840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077464.552737:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077464.552741:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077464.552742:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077464.552746:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4450 00000100:00080000:1.0:1769077464.552749:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077464.552753:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077464.552764:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077464.552776:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616739968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077464.552785:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616739840:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077464.552792:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077464.552796:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077464.552797:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077464.552801:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4450 00000100:00080000:1.0:1769077464.552804:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077464.552809:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077464.552814:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077464.552823:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616740096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077464.552830:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077464.552833:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077464.552835:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077464.552838:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4450 00000100:00080000:1.0:1769077464.552849:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077464.552852:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077464.552857:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077464.552870:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616740224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077464.552882:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077464.552887:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077464.552889:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077464.552894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739968:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077464.552899:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4450 00000100:00080000:1.0:1769077464.552907:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077464.552913:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077464.552920:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077464.552937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077464.552941:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616740352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077464.552956:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077464.552959:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077464.552961:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077464.552966:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4450 00000100:00080000:1.0:1769077464.552970:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077464.552975:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077464.552976:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740224:192.168.204.132@tcp:8:kworker.0 00000008:00100000:1.0:1769077464.552980:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077464.553002:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616740480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077464.553014:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077464.553017:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077464.553019:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077464.553024:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4450 00000100:00080000:1.0:1769077464.553030:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077464.553033:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077464.553037:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077464.553038:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077464.553052:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616740608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077464.553079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616740480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077464.553099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616740608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077465.256572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077465.256642:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077465.576586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077465.576606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077465.576627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077465.576640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077466.280559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077466.280632:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077466.600537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077466.600537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077466.600585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077466.600594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077467.304525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077467.304581:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077467.624566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077467.624566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077467.624567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077467.624619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769077467.624757:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769077467.624795:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077467.624811:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077467.624851:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077467.624860:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077467.624870:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077467.624878:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077467.624889:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077467.624901:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769077467.624903:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077464/real 1769077467] req@ffff995b868c4a80 x1855007616740352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077467.624932:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616740352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077467.624948:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077467.624954:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077467.624959:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077467.624968:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077464/real 1769077467] req@ffff995b868c6d80 x1855007616740480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077467.624982:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616740480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077467.624991:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077467.624995:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077467.624999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616740480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077467.625005:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077464/real 1769077467] req@ffff995b868c6300 x1855007616739968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077467.625018:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616739968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077467.625025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077467.625029:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077467.625032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616739968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077467.625039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077464/real 1769077467] req@ffff995b868c4700 x1855007616740096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077467.625051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616740096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077467.625062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077467.625065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077467.625068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077467.625075:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077464/real 1769077467] req@ffff995b868c5500 x1855007616740224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077467.625086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616740224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077467.625095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077467.625098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077467.625101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740224:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077467.625108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077464/real 1769077467] req@ffff995b868c7800 x1855007616739840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077467.625121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616739840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077467.625129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077467.625133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077467.625137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616739840:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077467.625173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077464/real 1769077467] req@ffff995b868c7480 x1855007616740608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077467.625189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616740608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077467.625198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077467.625202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077467.625206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616740608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077468.328561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077468.328627:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077468.648537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077468.648537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077468.648545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077468.648591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077469.352544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077469.352592:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077469.672572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077469.672572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077469.672572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077469.672608:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077469.672612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077469.672615:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077469.672621:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077469.672631:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4455 00000100:00080000:1.0:1769077469.672643:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077469.672651:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077469.672692:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616740736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077469.672733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077469.672737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077469.672739:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077469.672744:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4455 00000100:00080000:1.0:1769077469.672749:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077469.672754:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077469.672764:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077469.672779:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616740864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077469.672789:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077469.672792:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077469.672794:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077469.672798:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4455 00000100:00080000:1.0:1769077469.672801:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077469.672805:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077469.672810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616740736:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769077469.672810:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077469.672821:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616740992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077469.672830:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077469.672833:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077469.672834:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077469.672837:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4455 00000100:00080000:1.0:1769077469.672840:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077469.672844:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077469.672848:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077469.672857:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616741120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077469.672865:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077469.672868:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077469.672869:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077469.672873:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4455 00000100:00080000:1.0:1769077469.672876:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077469.672877:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077469.672880:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077469.672884:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077469.672893:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616741248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077469.672901:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077469.672904:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077469.672906:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077469.672910:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4455 00000100:00080000:1.0:1769077469.672913:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077469.672915:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077469/real 1769077469] req@ffff995b868c7100 x1855007616740736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077469.672917:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077469.672922:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077469.672931:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616741376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077469.672933:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616740736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077469.672942:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077469.672945:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077469.672946:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077469.672946:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077469.672950:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4455 00000100:00080000:0.0:1769077469.672951:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077469.672954:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077469.672955:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616740736:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077469.672958:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077469.672963:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740864:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769077469.672963:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077469.672972:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616741504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077469.672990:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077469.673004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077469/real 1769077469] req@ffff995b868c6a00 x1855007616740864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077469.673017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616740864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077469.673027:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077469.673031:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077469.673035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077469.673062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077469.673083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077469.673094:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077469/real 1769077469] req@ffff995b868c5c00 x1855007616740992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077469.673106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616740992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077469.673118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077469.673122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077469.673125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616740992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077469.673131:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616741120:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077469.673151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077469.673161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077469/real 1769077469] req@ffff995b868c7480 x1855007616741120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077469.673171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616741120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077469.673179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077469.673182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077469.673185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616741120:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077469.673191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616741248:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077469.673206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077469.673216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077469/real 1769077469] req@ffff995b868c7800 x1855007616741248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077469.673227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616741248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077469.673236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077469.673240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077469.673243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616741248:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077469.673248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616741376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077469.673284:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077469.673299:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077469/real 1769077469] req@ffff995b868c5500 x1855007616741376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077469.673311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616741376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077469.673320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077469.673323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077469.673340:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616741376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077469.673346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616741504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077469.673366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077469.673378:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077469/real 1769077469] req@ffff995b868c4700 x1855007616741504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077469.673390:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616741504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077469.673414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077469.673418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077469.673421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616741504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077470.376569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077470.376637:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077470.696555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077470.696575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077470.696582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077470.696706:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077471.400551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077471.400649:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077471.720582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077471.720582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077471.720608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077471.720637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077472.424558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077472.424615:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077472.744546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077472.744546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077472.744566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077472.744594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077473.448539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077473.448615:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077473.768546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077473.768547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077473.768551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077473.768592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077474.472572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077474.472641:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077474.792535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077474.792547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077474.792547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077474.792579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077474.792580:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077474.792587:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077474.792592:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077474.792601:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4460 00000100:00080000:1.0:1769077474.792613:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077474.792622:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077474.792665:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616741632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077474.792690:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077474.792694:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077474.792696:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077474.792701:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4460 00000100:00080000:1.0:1769077474.792706:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077474.792712:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077474.792723:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077474.792734:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616741632:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077474.792739:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616741760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077474.792753:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077474.792757:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077474.792759:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077474.792764:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4460 00000100:00080000:1.0:1769077474.792768:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077474.792772:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077474.792778:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077474.792792:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616741888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077474.792809:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077474.792812:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077474.792815:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077474.792818:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077474.792822:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4460 00000100:00080000:1.0:1769077474.792826:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077474.792830:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077474.792836:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077474.792848:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616742016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077474.792855:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077474.792857:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077474.792860:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077474.792861:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077474/real 1769077474] req@ffff995b868c6300 x1855007616741632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077474.792865:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4460 00000100:00080000:1.0:1769077474.792870:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077474.792876:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077474.792882:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077474.792890:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616741632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077474.792891:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616742144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077474.792920:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077474.792921:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077474.792926:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077474.792930:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769077474.792933:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616741632:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077474.792935:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077474.792943:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4460 00000100:00080000:1.0:1769077474.792949:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077474.792955:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077474.792959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616741760:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769077474.792962:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077474.792979:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616742272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077474.792988:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077474.792993:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077474.792994:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077474.792998:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077474.793003:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4460 00000100:00080000:1.0:1769077474.793007:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077474.793010:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077474/real 1769077474] req@ffff995b868c6d80 x1855007616741760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077474.793012:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077474.793017:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077474.793029:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616742400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077474.793037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616741760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077474.793051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077474.793056:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077474.793060:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616741760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077474.793068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616741888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077474.793087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077474.793095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077474/real 1769077474] req@ffff995b868c4a80 x1855007616741888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077474.793103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616741888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077474.793111:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077474.793114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077474.793116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616741888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077474.793121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742016:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077474.793131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077474.793137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077474/real 1769077474] req@ffff995b868c4700 x1855007616742016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077474.793150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616742016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077474.793156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077474.793157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077474.793159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742016:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077474.793163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742144:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077474.793172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077474.793178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077474/real 1769077474] req@ffff995b868c5500 x1855007616742144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077474.793185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616742144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077474.793194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077474.793198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077474.793201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742144:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077474.793220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616742272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077474.793242:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077474.793253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077474/real 1769077474] req@ffff995b868c7800 x1855007616742272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077474.793267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616742272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077474.793281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077474.793284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077474.793287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616742272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077474.793293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616742400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077474.793343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077474.793355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077474/real 1769077474] req@ffff995b868c7480 x1855007616742400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077474.793367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616742400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077474.793374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077474.793378:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077474.793380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616742400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077475.496538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077475.496599:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077475.816558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077475.816558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077475.816584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077475.816600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077476.520555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077476.520630:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077476.840543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077476.840561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077476.840564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077476.840565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077477.544524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077477.544607:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077477.864543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077477.864548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077477.864552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077477.864589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077478.568560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077478.568626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077478.888544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077478.888556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077478.888582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077478.888590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077479.592531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077479.592962:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077479.912561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077479.912578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077479.912596:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077479.912604:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077479.912610:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077479.912620:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4465 00000100:00080000:2.0:1769077479.912633:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077479.912641:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077479.912645:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077479.912681:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616742528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077479.912714:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077479.912721:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077479.912723:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077479.912728:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4465 00000100:00080000:2.0:1769077479.912732:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077479.912737:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077479.912738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616742528:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077479.912746:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077479.912759:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616742656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077479.912770:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077479.912773:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077479.912775:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077479.912779:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4465 00000100:00080000:2.0:1769077479.912782:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077479.912785:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077479.912798:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077479.912802:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077479.912814:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616742784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077479.912823:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077479.912826:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077479.912828:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077479.912831:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4465 00000100:00080000:2.0:1769077479.912835:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077479.912839:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077479.912840:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077479/real 1769077479] req@ffff995b85acbb80 x1855007616742528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077479.912845:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077479.912854:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616742912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077479.912859:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616742528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077479.912865:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077479.912868:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077479.912870:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077479.912870:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077479.912874:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4465 00000100:00080000:0.0:1769077479.912875:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077479.912879:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616742528:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077479.912887:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077479.912890:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077479.912895:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077479.912903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077479.912904:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616743040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077479.912914:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077479.912918:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077479.912920:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769077479.912925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077479.912925:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4465 00000100:00080000:2.0:1769077479.912928:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077479.912932:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077479.912937:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077479/real 1769077479] req@ffff995b85ac8380 x1855007616742656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077479.912937:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077479.912947:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616743168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077479.912951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616742656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077479.912957:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077479.912960:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077479.912961:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077479.912963:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077479.912964:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077479.912967:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4465 00000100:00100000:0.0:1769077479.912968:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742656:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077479.912970:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077479.912974:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742784:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077479.912974:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077479.912978:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077479.912987:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616743296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077479.912992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077479.913002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077479/real 1769077479] req@ffff995b85ac9880 x1855007616742784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077479.913016:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616742784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077479.913034:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077479.913037:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077479.913040:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077479.913044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742912:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077479.913069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077479.913077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077479/real 1769077479] req@ffff995b85ac9c00 x1855007616742912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077479.913085:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616742912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077479.913091:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077479.913093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077479.913095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616742912:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077479.913107:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743040:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077479.913124:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077479.913135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077479/real 1769077479] req@ffff995b85acb800 x1855007616743040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077479.913147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616743040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077479.913157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077479.913161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077479.913165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077479.913171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616743168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077479.913188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077479.913199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077479/real 1769077479] req@ffff995b85aca300 x1855007616743168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077479.913212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616743168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077479.913236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077479.913241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077479.913245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616743168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077479.913258:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616743296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077479.913280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077479.913290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077479/real 1769077479] req@ffff995b85acaa00 x1855007616743296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077479.913299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616743296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077479.913304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077479.913306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077479.913309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616743296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077479.913525:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077480.616550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077480.616723:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077480.936556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077480.936585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077480.936739:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077480.936904:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077481.640510:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077481.640611:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077481.960534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077481.960534:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077481.960538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077481.960546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077482.664529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077482.664610:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077482.984564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077482.984564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077482.984592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077482.984612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077483.688548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077483.688607:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077484.008541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077484.008547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077484.008547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077484.008576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077484.712540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077484.712610:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077485.032554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077485.032559:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077485.032566:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077485.032571:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077485.032580:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4470 00000100:00100000:2.0:1769077485.032582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077485.032591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077485.032592:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077485.032602:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077485.032606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077485.032641:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616743424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077485.032662:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077485.032665:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077485.032666:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077485.032670:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4470 00000100:00080000:3.0:1769077485.032673:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077485.032678:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077485.032689:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077485.032703:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616743424:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077485.032706:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616743552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077485.032721:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077485.032725:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077485.032726:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077485.032730:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4470 00000100:00080000:3.0:1769077485.032733:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077485.032736:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077485.032741:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077485.032753:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616743680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077485.032764:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077485.032769:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077485.032771:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077485.032775:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077485.032781:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4470 00000100:00080000:3.0:1769077485.032787:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077485.032792:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077485.032799:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077485.032808:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077485/real 1769077485] req@ffff995b8455aa00 x1855007616743424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077485.032823:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616743808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077485.032825:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616743424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077485.032854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077485.032858:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077485.032860:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077485.032864:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077485.032868:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077485.032872:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4470 00000100:00080000:3.0:1769077485.032877:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077485.032881:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616743424:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077485.032883:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077485.032891:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077485.032902:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616743936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077485.032909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077485.032911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077485.032913:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077485.032917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743552:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077485.032926:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4470 00000100:00080000:3.0:1769077485.032930:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077485.032933:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077485.032937:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077485.032943:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077485.032948:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616744064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077485.032957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077485.032959:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077485/real 1769077485] req@ffff995b8455bb80 x1855007616743552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077485.032960:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077485.032963:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077485.032970:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4470 00000100:00080000:3.0:1769077485.032975:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077485.032979:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077485.032983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616743552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077485.032985:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077485.032997:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077485.033001:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616744192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077485.033003:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077485.033009:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077485.033030:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077485.033056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077485.033069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077485/real 1769077485] req@ffff995b84559500 x1855007616743680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077485.033082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616743680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077485.033090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077485.033093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077485.033095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077485.033100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743808:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077485.033116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077485.033127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077485/real 1769077485] req@ffff995b84558a80 x1855007616743808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077485.033155:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616743808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077485.033165:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077485.033178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077485.033182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743808:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077485.033190:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743936:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077485.033207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077485.033215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077485/real 1769077485] req@ffff995b8455a680 x1855007616743936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077485.033229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616743936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077485.033236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077485.033238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077485.033240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616743936:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077485.033255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616744064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077485.033266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077485.033273:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077485/real 1769077485] req@ffff995b8455b100 x1855007616744064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077485.033281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616744064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077485.033286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077485.033288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077485.033290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616744064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077485.033294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616744192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077485.033303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077485.033309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077485/real 1769077485] req@ffff995b8455aa00 x1855007616744192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077485.033316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616744192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077485.033326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077485.033328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077485.033330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616744192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077485.736550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077485.736652:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077486.056533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077486.056552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077486.056564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077486.056574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077486.760533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077486.760591:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077487.080566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077487.080584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077487.080600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077487.080619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077487.784554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077487.784625:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077488.104558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077488.104562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077488.104680:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077488.104685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077488.808547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077488.808604:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077489.128537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077489.128586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077489.128621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077489.128704:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077489.832563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077489.832662:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077490.152539:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077490.152548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077490.152549:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077490.152555:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077490.152557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077490.152564:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4475 00000100:00100000:1.0:1769077490.152570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077490.152576:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077490.152584:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077490.152586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077490.152622:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616744320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077490.152642:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077490.152645:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077490.152646:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077490.152649:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4475 00000100:00080000:3.0:1769077490.152651:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077490.152654:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077490.152662:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077490.152671:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616744448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077490.152678:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077490.152681:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616744320:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077490.152682:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077490.152686:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077490.152694:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4475 00000100:00080000:3.0:1769077490.152696:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077490.152699:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077490.152702:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077490.152713:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616744576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077490.152722:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077490.152726:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077490.152728:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077490.152732:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4475 00000100:00080000:3.0:1769077490.152736:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077490.152740:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077490.152746:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077490.152759:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616744704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077490.152768:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077490.152775:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077490.152779:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077490.152781:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077490.152786:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4475 00000100:00080000:3.0:1769077490.152790:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077490.152794:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077490.152799:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077490.152816:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077490/real 1769077490] req@ffff995b84558000 x1855007616744320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077490.152816:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616744832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077490.152848:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077490.152853:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077490.152855:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077490.152859:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616744320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077490.152863:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4475 00000100:00080000:3.0:1769077490.152872:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077490.152876:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077490.152879:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769077490.152885:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769077490.152887:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077490.152893:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616744320:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077490.152901:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616744960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077490.152909:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077490.152912:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077490.152913:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077490.152916:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4475 00000100:00080000:3.0:1769077490.152919:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077490.152922:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077490.152926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616744448:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769077490.152927:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077490.152948:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616745088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077490.152961:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077490.152979:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077490/real 1769077490] req@ffff995b84559880 x1855007616744448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077490.152993:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616744448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077490.153001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077490.153004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077490.153008:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616744448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077490.153014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616744576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077490.153034:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077490.153046:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077490/real 1769077490] req@ffff995b84559c00 x1855007616744576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077490.153074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616744576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077490.153081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077490.153083:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077490.153086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616744576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077490.153091:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616744704:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077490.153106:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077490.153113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077490/real 1769077490] req@ffff995b8455a300 x1855007616744704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077490.153121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616744704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077490.153132:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077490.153134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077490.153136:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616744704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077490.153139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616744832:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077490.153148:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077490.153154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077490/real 1769077490] req@ffff995b84558700 x1855007616744832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077490.153162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616744832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077490.153168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077490.153169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077490.153171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616744832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077490.153175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616744960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077490.153184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077490.153190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077490/real 1769077490] req@ffff995b8455b480 x1855007616744960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077490.153202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616744960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077490.153208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077490.153210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077490.153213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616744960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077490.153228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616745088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077490.153239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077490.153246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077490/real 1769077490] req@ffff995b8455ad80 x1855007616745088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077490.153256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616745088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077490.153265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077490.153269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077490.153273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616745088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077490.856513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077490.856592:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077491.176540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077491.176558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077491.176579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077491.176596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077491.880576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077491.880631:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077492.200519:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077492.200537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077492.200539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077492.200564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077492.904615:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077492.904646:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077493.224579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077493.224579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077493.224580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077493.224601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077493.928557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077493.928615:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077494.248538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077494.248538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077494.248549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077494.248552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077494.952537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077494.952595:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077495.272537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077495.272543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077495.272559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077495.272588:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077495.272593:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077495.272597:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077495.272605:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4480 00000100:00080000:1.0:1769077495.272616:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077495.272624:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077495.272645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077495.272662:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616745216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077495.272696:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077495.272701:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077495.272703:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077495.272708:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4480 00000100:00080000:1.0:1769077495.272713:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077495.272719:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077495.272731:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077495.272737:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616745216:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077495.272751:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616745344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077495.272767:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077495.272772:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077495.272774:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077495.272779:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4480 00000100:00080000:1.0:1769077495.272784:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077495.272787:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077495.272792:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077495.272804:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616745472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077495.272814:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077495.272814:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077495.272820:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077495.272822:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077495.272826:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4480 00000100:00080000:1.0:1769077495.272830:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077495.272832:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077495.272836:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077495.272855:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616745600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077495.272864:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077495.272866:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077495/real 1769077495] req@ffff995b868c5c00 x1855007616745216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077495.272867:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077495.272870:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077495.272878:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4480 00000100:00080000:1.0:1769077495.272884:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077495.272890:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077495.272894:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077495.272903:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616745216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077495.272904:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616745728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077495.272935:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077495.272939:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077495.272941:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077495.272941:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077495.272951:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077495.272956:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4480 00000100:00100000:2.0:1769077495.272961:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616745216:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077495.272975:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077495.272984:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077495.272990:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077495.273005:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616745856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077495.273009:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616745344:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077495.273027:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077495.273032:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077495.273034:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077495.273040:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4480 00000800:00000100:2.0:1769077495.273042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077495.273048:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077495.273056:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077495.273060:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077495/real 1769077495] req@ffff995b868c6a00 x1855007616745344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077495.273063:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077495.273082:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616745984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077495.273083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616745344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077495.273115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077495.273119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077495.273123:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616745344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077495.273129:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616745472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077495.273149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077495.273163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077495/real 1769077495] req@ffff995b868c7100 x1855007616745472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077495.273176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616745472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077495.273186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077495.273190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077495.273194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616745472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077495.273200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616745600:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077495.273221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077495.273234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077495/real 1769077495] req@ffff995b868c7480 x1855007616745600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077495.273247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616745600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077495.273258:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077495.273261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077495.273265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616745600:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077495.273272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616745728:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077495.273292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077495.273300:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077495/real 1769077495] req@ffff995b868c7800 x1855007616745728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077495.273309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616745728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077495.273324:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077495.273327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077495.273329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616745728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077495.273342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616745856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077495.273358:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077495.273367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077495/real 1769077495] req@ffff995b868c5500 x1855007616745856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077495.273388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616745856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077495.273394:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077495.273396:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077495.273398:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616745856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077495.273403:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616745984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077495.273415:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077495.273422:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077495/real 1769077495] req@ffff995b868c4700 x1855007616745984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077495.273429:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616745984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077495.273439:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077495.273441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077495.273443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616745984:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769077495.976615:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077495.976653:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077496.296544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077496.296566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077496.296567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077496.296599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077497.000525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077497.000581:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077497.320536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077497.320536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077497.320551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077497.320578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077498.024549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077498.024618:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077498.344535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077498.344547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077498.344547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077498.344548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077499.048529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077499.048601:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077499.368576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077499.368576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077499.368576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077499.368597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077500.072608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077500.072689:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077500.392653:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077500.392654:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077500.392666:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077500.392672:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077500.392678:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077500.392685:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4486 00000100:00080000:3.0:1769077500.392707:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077500.392718:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077500.392789:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616746112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077500.392815:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077500.392820:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077500.392822:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077500.392827:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4486 00000100:00080000:3.0:1769077500.392832:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077500.392837:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077500.392850:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077500.392851:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616746112:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077500.392870:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616746240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077500.392894:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077500.392896:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077500.392901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077500.392908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077500.392911:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077500.392920:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4486 00000100:00080000:3.0:1769077500.392934:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077500.392940:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077500.392947:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077500.392951:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077500.392965:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616746368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077500.392975:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077500.392977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077500.392979:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077500.392983:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4486 00000100:00080000:3.0:1769077500.392987:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077500.392991:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077500.392996:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077500.392999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077500/real 1769077500] req@ffff995b8455ad80 x1855007616746112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077500.393008:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616746496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077500.393016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077500.393019:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616746112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077500.393021:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077500.393024:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077500.393031:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4486 00000100:00080000:3.0:1769077500.393036:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077500.393040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077500.393041:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077500.393048:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769077500.393050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077500.393056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616746112:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077500.393059:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616746624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077500.393068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077500.393070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077500.393073:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077500.393076:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4486 00000100:00080000:3.0:1769077500.393078:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077500.393081:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077500.393085:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616746240:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769077500.393086:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077500.393099:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616746752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077500.393112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077500.393118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077500.393123:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077500.393126:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077500.393127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077500/real 1769077500] req@ffff995b8455b480 x1855007616746240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077500.393135:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4486 00000100:00080000:3.0:1769077500.393144:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077500.393147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616746240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077500.393153:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077500.393158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077500.393163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769077500.393163:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077500.393169:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616746240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077500.393175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616746368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077500.393190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077500.393193:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616746880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077500.393202:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077500/real 1769077500] req@ffff995b84558700 x1855007616746368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077500.393213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616746368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077500.393222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077500.393224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077500.393227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616746368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077500.393231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616746496:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077500.393242:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077500.393248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077500/real 1769077500] req@ffff995b8455a300 x1855007616746496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077500.393260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616746496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077500.393266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077500.393268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077500.393270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616746496:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077500.393273:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616746624:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077500.393283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077500.393289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077500/real 1769077500] req@ffff995b84559c00 x1855007616746624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077500.393296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616746624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077500.393302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077500.393303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077500.393305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616746624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077500.393316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616746752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077500.393326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077500.393332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077500/real 1769077500] req@ffff995b84559880 x1855007616746752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077500.393339:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616746752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077500.393349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077500.393351:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077500.393354:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616746752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077500.393357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616746880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077500.393366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077500.393371:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077500/real 1769077500] req@ffff995b84558000 x1855007616746880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077500.393379:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616746880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077500.393385:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077500.393387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077500.393389:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616746880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077501.096605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077501.096709:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077501.416601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077501.416609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077501.416636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077501.416648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077502.120548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077502.120621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077502.440568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077502.440568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077502.440606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077502.440622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077503.144564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077503.144651:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077503.464566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077503.464587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077503.464600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077503.464618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077504.168508:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077504.168561:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077504.488565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077504.488565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077504.488565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077504.488584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077505.192568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077505.192634:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077505.512543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077505.512543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077505.512550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077505.512551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077505.512611:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077505.512618:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077505.512624:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077505.512635:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4491 00000100:00080000:2.0:1769077505.512647:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077505.512654:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077505.512714:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616747008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077505.512737:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077505.512740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077505.512742:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077505.512745:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4491 00000100:00080000:2.0:1769077505.512748:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077505.512752:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077505.512761:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077505.512772:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616747136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077505.512779:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616747008:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077505.512782:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077505.512812:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077505.512814:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077505.512818:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4491 00000100:00080000:2.0:1769077505.512822:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077505.512827:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077505.512832:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077505.512849:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616747264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077505.512866:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077505.512872:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077505.512873:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077505.512878:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077505.512884:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4491 00000100:00080000:2.0:1769077505.512889:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077505.512893:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077505.512898:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077505.512912:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616747392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077505.512920:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077505/real 1769077505] req@ffff995b85ac8a80 x1855007616747008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077505.512928:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077505.512935:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077505.512937:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077505.512941:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616747008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077505.512946:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4491 00000100:00080000:2.0:1769077505.512956:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077505.512957:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077505.512965:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769077505.512966:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077505.512972:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616747008:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077505.512976:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077505.512991:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616747520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077505.512995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616747136:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077505.513009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077505.513013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077505.513015:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077505.513018:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4491 00000100:00080000:2.0:1769077505.513022:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077505.513026:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077505.513030:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077505.513033:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077505.513045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077505/real 1769077505] req@ffff995b85acb100 x1855007616747136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077505.513046:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616747648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077505.513078:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077505.513082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769077505.513083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616747136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077505.513084:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077505.513092:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4491 00000100:00080000:2.0:1769077505.513097:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077505.513101:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077505.513103:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769077505.513107:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077505.513111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616747136:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077505.513113:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077505.513117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616747264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077505.513132:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616747776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077505.513138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077505.513149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077505/real 1769077505] req@ffff995b85acaa00 x1855007616747264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077505.513159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616747264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077505.513167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077505.513170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077505.513172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616747264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077505.513176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616747392:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077505.513187:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077505.513193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077505/real 1769077505] req@ffff995b85aca300 x1855007616747392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077505.513200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616747392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077505.513206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077505.513208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077505.513210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616747392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077505.513220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616747520:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077505.513238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077505.513245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077505/real 1769077505] req@ffff995b85acb800 x1855007616747520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077505.513257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616747520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077505.513263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077505.513266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077505.513268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616747520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077505.513272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616747648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077505.513282:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077505.513288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077505/real 1769077505] req@ffff995b85ac9c00 x1855007616747648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077505.513295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616747648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077505.513301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077505.513302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077505.513304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616747648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077505.513308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616747776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077505.513317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077505.513323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077505/real 1769077505] req@ffff995b85ac9880 x1855007616747776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077505.513330:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616747776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077505.513340:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077505.513342:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077505.513344:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616747776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077506.216560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077506.216641:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077506.536561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077506.536561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077506.536561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077506.536561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077507.240572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077507.240629:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077507.560543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077507.560549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077507.560566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077507.560580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077508.264517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077508.264591:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077508.584547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077508.584547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077508.584571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077508.584577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077509.288555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077509.288619:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077509.608530:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077509.608551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077509.608602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077509.608610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077510.312513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077510.312569:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077510.632592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077510.632592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077510.632600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077510.632605:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077510.632614:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077510.632620:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077510.632630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077510.632631:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4496 00000100:00080000:0.0:1769077510.632662:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077510.632671:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077510.632731:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616747904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077510.632760:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077510.632765:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077510.632767:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077510.632771:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4496 00000100:00080000:0.0:1769077510.632776:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077510.632780:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077510.632789:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077510.632798:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616747904:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077510.632803:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616748032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077510.632822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077510.632824:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077510.632826:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077510.632828:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4496 00000100:00080000:0.0:1769077510.632831:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077510.632833:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077510.632837:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077510.632846:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616748160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077510.632852:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077510.632854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077510.632855:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077510.632857:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4496 00000100:00080000:0.0:1769077510.632860:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077510.632862:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077510.632866:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077510.632874:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616748288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077510.632878:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077510.632887:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077510.632890:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077510.632892:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077510.632895:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4496 00000100:00080000:0.0:1769077510.632899:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077510.632901:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077510.632904:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077510.632912:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616748416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077510.632917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077510.632919:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077510/real 1769077510] req@ffff995b84640380 x1855007616747904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077510.632921:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077510.632924:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077510.632931:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4496 00000100:00080000:0.0:1769077510.632936:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077510.632940:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077510.632943:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077510.632952:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616748544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077510.632957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616747904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077510.632960:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077510.632966:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077510.632968:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077510.632975:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4496 00000100:00080000:0.0:1769077510.632979:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077510.632983:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077510.632986:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077510.632986:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077510.632998:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616748672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077510.632998:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077510.633011:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616747904:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077510.633048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077510.633073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077510.633086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077510/real 1769077510] req@ffff995b84641f80 x1855007616748032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077510.633098:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616748032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077510.633107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077510.633110:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077510.633114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077510.633120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077510.633138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077510.633149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077510/real 1769077510] req@ffff995b84640000 x1855007616748160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077510.633169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616748160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077510.633187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077510.633191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077510.633194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077510.633201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748288:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077510.633222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077510.633234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077510/real 1769077510] req@ffff995b84640a80 x1855007616748288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077510.633247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616748288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077510.633257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077510.633261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077510.633272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077510.633280:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748416:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077510.633300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077510.633312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077510/real 1769077510] req@ffff995b84641c00 x1855007616748416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077510.633324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616748416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077510.633341:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077510.633345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077510.633348:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077510.633355:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616748544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077510.633373:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077510.633384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077510/real 1769077510] req@ffff995b84640e00 x1855007616748544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077510.633406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616748544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077510.633415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077510.633419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077510.633423:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616748544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077510.633429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616748672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077510.633448:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077510.633475:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077510/real 1769077510] req@ffff995b84642680 x1855007616748672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077510.633484:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616748672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077510.633489:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077510.633491:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077510.633494:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616748672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077511.336556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077511.336592:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077511.656558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077511.656558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077511.656583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077511.656600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077512.360561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077512.360594:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077512.680561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077512.680573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077512.680591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077512.680597:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077513.384570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077513.384670:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077513.704557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077513.704557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077513.704562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077513.704812:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077514.408548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077514.408610:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077514.728557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077514.728557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077514.728577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077514.728580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077515.432570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077515.432659:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077515.752539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077515.752561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077515.752577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077515.752584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077515.752645:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077515.752653:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077515.752659:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077515.752671:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4501 00000100:00080000:3.0:1769077515.752684:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077515.752692:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077515.752734:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616748800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077515.752762:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077515.752767:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077515.752769:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077515.752774:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4501 00000100:00080000:3.0:1769077515.752779:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077515.752784:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077515.752796:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077515.752804:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616748800:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077515.752813:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616748928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077515.752826:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077515.752838:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077515.752841:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077515.752847:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4501 00000100:00080000:3.0:1769077515.752864:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077515.752870:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077515.752876:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077515.752883:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077515.752895:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616749056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077515.752909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077515.752913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077515.752915:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077515.752920:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4501 00000100:00100000:2.0:1769077515.752926:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077515/real 1769077515] req@ffff995b8455aa00 x1855007616748800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077515.752927:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077515.752937:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077515.752944:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077515.752948:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616748800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077515.752958:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077515.752962:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616749184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077515.752965:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077515.752971:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616748800:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077515.752986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077515.752990:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077515.752992:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077515.752998:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748928:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077515.752999:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4501 00000100:00080000:3.0:1769077515.753019:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077515.753022:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077515.753028:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077515.753035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077515/real 1769077515] req@ffff995b8455b100 x1855007616748928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077515.753036:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077515.753051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616748928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077515.753059:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616749312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077515.753062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077515.753069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077515.753075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616748928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077515.753085:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749056:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077515.753085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077515.753093:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1769077515.753101:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077515.753104:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077515.753111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077515/real 1769077515] req@ffff995b8455a680 x1855007616749056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077515.753113:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4501 00000100:00080000:3.0:1769077515.753123:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077515.753127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616749056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077515.753133:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077515.753139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769077515.753143:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077515.753144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077515.753150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077515.753156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077515.753165:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616749440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077515.753172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077515.753182:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077515.753183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077515/real 1769077515] req@ffff995b84558a80 x1855007616749184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077515.753193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077515.753199:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077515.753201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616749184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077515.753208:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4501 00000100:00080000:2.0:1769077515.753212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077515.753216:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077515.753217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077515.753223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749184:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077515.753226:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077515.753235:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077515.753238:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077515.753254:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616749568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077515.753256:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077515.753266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077515/real 1769077515] req@ffff995b84559500 x1855007616749312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077515.753280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616749312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077515.753287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077515.753290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077515.753292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077515.753297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616749440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077515.753307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077515.753313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077515/real 1769077515] req@ffff995b8455bb80 x1855007616749440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077515.753321:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616749440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077515.753328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077515.753331:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077515.753334:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616749440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077515.753346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616749568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077515.753366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077515.753378:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077515/real 1769077515] req@ffff995b84558a80 x1855007616749568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077515.753391:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616749568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077515.753397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077515.753399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077515.753401:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616749568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077516.456554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077516.456647:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077516.776557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077516.776563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077516.776596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077516.776642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077517.480566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077517.480655:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077517.800629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077517.800638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077517.800644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077517.800674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077518.504599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077518.504614:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077518.824626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077518.824626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077518.824653:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077518.824653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077519.528545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077519.528640:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077519.848650:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077519.848650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077519.848651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077519.848680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077520.552589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077520.552648:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077520.872583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077520.872611:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077520.872618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077520.872624:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077520.872635:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077520.872636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077520.872655:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4506 00000100:00100000:2.0:1769077520.872667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077520.872672:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077520.872683:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077520.872731:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616749696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077520.872766:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077520.872770:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077520.872773:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077520.872778:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4506 00000100:00080000:1.0:1769077520.872785:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077520.872791:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077520.872803:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077520.872813:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616749696:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077520.872824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616749824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077520.872839:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077520.872844:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077520.872847:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077520.872852:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4506 00000100:00080000:1.0:1769077520.872857:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077520.872863:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077520.872869:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077520.872884:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616749952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077520.872897:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077520.872914:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077520.872919:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077520.872931:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077520.872938:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4506 00000100:00100000:2.0:1769077520.872940:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077520/real 1769077520] req@ffff995b868c4a80 x1855007616749696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077520.872947:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077520.872959:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077520.872966:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616749696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077520.872972:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769077520.872976:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077520.872983:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077520.872987:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616749696:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077520.872994:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616750080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077520.873009:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077520.873014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749824:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077520.873016:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077520.873025:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077520.873033:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4506 00000100:00080000:1.0:1769077520.873041:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077520.873042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077520.873054:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077520.873058:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077520/real 1769077520] req@ffff995b868c6d80 x1855007616749824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077520.873064:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077520.873072:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616749824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077520.873082:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077520.873088:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616750208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077520.873092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077520.873097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077520.873104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749952:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077520.873129:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077520.873134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077520.873142:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077520.873145:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077520.873153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077520/real 1769077520] req@ffff995b868c6300 x1855007616749952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077520.873158:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4506 00000100:00080000:1.0:1769077520.873167:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077520.873177:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077520.873185:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077520.873186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616749952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077520.873206:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616750336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077520.873211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077520.873224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077520.873233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616749952:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077520.873245:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077520.873249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750080:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769077520.873256:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077520.873261:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077520.873268:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4506 00000100:00080000:1.0:1769077520.873277:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077520.873281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077520.873288:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077520.873295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077520/real 1769077520] req@ffff995b868c4700 x1855007616750080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077520.873299:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077520.873314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616750080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077520.873324:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077520.873326:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616750464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077520.873329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077520.873338:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077520.873372:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750208:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077520.873386:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077520.873397:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077520/real 1769077520] req@ffff995b868c5500 x1855007616750208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077520.873404:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616750208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077520.873411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077520.873413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077520.873414:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750208:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077520.873418:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616750336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077520.873429:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077520.873435:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077520/real 1769077520] req@ffff995b868c7800 x1855007616750336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077520.873447:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616750336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077520.873470:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077520.873473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077520.873475:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616750336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077520.873479:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616750464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077520.873492:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077520.873499:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077520/real 1769077520] req@ffff995b868c7480 x1855007616750464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077520.873507:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616750464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077520.873512:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077520.873515:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077520.873517:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616750464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077521.576585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077521.576674:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077521.896622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077521.896622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077521.896622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077521.896647:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077522.600566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077522.600631:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077522.920585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077522.920585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077522.920585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077522.920585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077523.624570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077523.624641:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077523.944618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077523.944625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077523.944627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077523.944645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077524.648586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077524.648681:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077524.968560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077524.968560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077524.968562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077524.968565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077525.672550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077525.672637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077525.992585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077525.992585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077525.992595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077525.992613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077525.992626:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077525.992633:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077525.992638:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077525.992649:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4511 00000100:00080000:2.0:1769077525.992662:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077525.992670:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077525.992713:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616750592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077525.992738:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077525.992741:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077525.992742:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077525.992746:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4511 00000100:00080000:2.0:1769077525.992751:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077525.992757:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077525.992767:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077525.992776:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616750592:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077525.992786:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616750720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077525.992799:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077525.992803:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077525.992805:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077525.992809:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4511 00000100:00080000:2.0:1769077525.992812:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077525.992815:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077525.992821:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077525.992831:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616750848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077525.992838:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077525.992841:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077525.992842:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077525.992844:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4511 00000100:00080000:2.0:1769077525.992846:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077525.992848:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077525.992851:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077525.992857:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616750976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077525.992863:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077525.992865:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077525.992866:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077525.992869:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4511 00000100:00080000:2.0:1769077525.992872:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077525.992875:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077525.992877:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077525.992882:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077525.992889:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616751104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077525.992896:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077525.992899:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077525.992901:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077525.992903:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4511 00000100:00080000:2.0:1769077525.992905:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077525.992907:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077525.992910:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077525.992916:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616751232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077525.992921:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077525.992923:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077525.992924:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077525.992926:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4511 00000100:00080000:2.0:1769077525.992928:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077525.992931:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077525.992935:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077525.992941:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077525/real 1769077525] req@ffff995b85ac8380 x1855007616750592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077525.992945:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616751360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077525.992996:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616750592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077525.993009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077525.993014:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077525.993019:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616750592:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077525.993048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077525.993084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077525.993098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077525/real 1769077525] req@ffff995b85acbb80 x1855007616750720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077525.993111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616750720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077525.993121:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077525.993125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077525.993128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077525.993135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077525.993151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077525.993171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077525/real 1769077525] req@ffff995b85ac9880 x1855007616750848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077525.993182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616750848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077525.993194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077525.993197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077525.993201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077525.993207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077525.993225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077525.993234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077525/real 1769077525] req@ffff995b85ac9c00 x1855007616750976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077525.993248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616750976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077525.993258:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077525.993261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077525.993265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616750976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077525.993271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616751104:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077525.993286:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077525.993295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077525/real 1769077525] req@ffff995b85acb800 x1855007616751104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077525.993307:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616751104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077525.993318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077525.993321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077525.993325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616751104:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077525.993330:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616751232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077525.993346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077525.993355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077525/real 1769077525] req@ffff995b85aca300 x1855007616751232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077525.993367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616751232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077525.993383:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077525.993387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077525.993391:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616751232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077525.993397:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616751360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077525.993412:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077525.993421:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077525/real 1769077525] req@ffff995b85acaa00 x1855007616751360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077525.993434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616751360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077525.993442:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077525.993479:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077525.993485:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616751360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077526.696558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077526.696620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077527.016545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077527.016545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077527.016546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077527.016566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077527.720532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077527.720589:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077528.040560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077528.040560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077528.040562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077528.040580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077528.744558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077528.744622:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077529.064546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077529.064553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077529.064554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077529.064575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077529.768527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077529.768610:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077530.088566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077530.088567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077530.088572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077530.088720:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077530.792600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077530.792690:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077531.112536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077531.112547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077531.112567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077531.112569:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077531.112578:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077531.112583:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077531.112595:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4516 00000100:00080000:2.0:1769077531.112609:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077531.112614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077531.112619:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077531.112665:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616751488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077531.112700:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077531.112704:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077531.112708:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077531.112714:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4516 00000100:00080000:2.0:1769077531.112719:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077531.112724:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077531.112731:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616751488:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077531.112735:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077531.112754:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616751616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077531.112768:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077531.112772:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077531.112775:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077531.112780:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4516 00000100:00080000:2.0:1769077531.112788:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077531.112798:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077531.112808:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077531.112826:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616751744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077531.112834:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616751616:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077531.112845:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077531.112850:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077531.112853:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077531.112859:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4516 00000100:00080000:2.0:1769077531.112868:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077531.112873:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077531.112878:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077531.112889:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616751872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077531.112895:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077531.112898:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077531.112900:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077531.112902:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4516 00000100:00080000:2.0:1769077531.112905:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077531.112907:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077531.112910:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077531.112918:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616752000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077531.112924:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077531.112926:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077531.112928:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077531.112932:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4516 00000100:00080000:2.0:1769077531.112936:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077531.112940:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616751744:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077531.112941:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077531.112950:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077531.112962:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616752128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077531.112969:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077531.112972:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077531.112973:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077531.112978:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4516 00000100:00080000:2.0:1769077531.112983:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077531.112986:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077531.112990:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077531.112999:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616752256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077531.113102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616751872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077531.113136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077531.113157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616752128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077531.113179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616752256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077531.816568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077531.816625:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077532.136584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077532.136586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077532.136689:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077532.136695:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077532.840568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077532.840642:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077533.160563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077533.160573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077533.160586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077533.160660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077533.864544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077533.864599:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077534.184551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077534.184551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077534.184559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077534.184827:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:3.0:1769077534.184962:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:3.0:1769077534.185015:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769077534.185035:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077534.185077:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077534.185087:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077534.185097:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077534.185107:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077534.185117:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077534.185128:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769077534.185132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077531/real 1769077534] req@ffff995b85acaa00 x1855007616751744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077534.185171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616751744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077534.185187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077534.185193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077534.185198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616751744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077534.185208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077531/real 1769077534] req@ffff995b85aca300 x1855007616751872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077534.185221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616751872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077534.185230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077534.185234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077534.185237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616751872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077534.185243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077531/real 1769077534] req@ffff995b85acb800 x1855007616752000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077534.185255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616752000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077534.185263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077534.185267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077534.185270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077534.185277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077531/real 1769077534] req@ffff995b85ac8a80 x1855007616751616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077534.185289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616751616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077534.185297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077534.185300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077534.185303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616751616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077534.185345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077531/real 1769077534] req@ffff995b85acb100 x1855007616751488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077534.185361:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616751488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077534.185369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077534.185373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077534.185377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616751488:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077534.185408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077531/real 1769077534] req@ffff995b85ac9c00 x1855007616752128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077534.185421:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616752128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077534.185430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077534.185434:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077534.185437:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616752128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077534.185443:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077531/real 1769077534] req@ffff995b85ac9880 x1855007616752256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077534.185479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616752256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077534.185488:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077534.185492:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077534.185495:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616752256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077534.888564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077534.888688:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077535.208543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077535.208543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077535.208554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077535.208658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077535.912586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077535.912662:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077536.232586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077536.232587:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077536.232592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077536.232597:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077536.232603:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077536.232615:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4521 00000100:00080000:2.0:1769077536.232628:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077536.232636:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077536.232644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077536.232690:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616752384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077536.232720:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077536.232724:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077536.232726:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077536.232732:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4521 00000100:00080000:2.0:1769077536.232738:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077536.232746:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077536.232758:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077536.232780:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616752512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077536.232788:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077536.232795:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077536.232800:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077536.232804:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077536.232822:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4521 00000100:00100000:1.0:1769077536.232825:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616752384:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077536.232827:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077536.232836:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077536.232848:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077536.232871:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616752640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077536.232899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077536.232904:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077536.232906:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077536.232912:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4521 00000100:00080000:2.0:1769077536.232918:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077536.232928:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077536.232936:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769077536.232943:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077536.232952:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616752768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077536.232968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077536.232973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077536.232975:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077536.232980:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4521 00000100:00080000:2.0:1769077536.232985:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077536.232991:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077536.232998:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077536.233004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077536/real 1769077536] req@ffff995b85acbb80 x1855007616752384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077536.233020:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616752896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077536.233037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616752384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077536.233081:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077536.233086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077536.233089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077536.233095:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077536.233097:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077536.233115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616752384:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077536.233116:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4521 00000100:00080000:2.0:1769077536.233134:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077536.233144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752512:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077536.233144:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077536.233164:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077536.233191:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616753024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077536.233203:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077536.233216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077536.233217:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077536.233224:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077536.233236:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4521 00000100:00080000:2.0:1769077536.233244:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077536.233246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077536/real 1769077536] req@ffff995b85ac8380 x1855007616752512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077536.233257:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077536.233288:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077536.233311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616752512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077536.233318:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616753152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077536.233360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077536.233371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077536.233376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077536.233413:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077536.233439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077536.233451:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077536/real 1769077536] req@ffff995b85ac9880 x1855007616752640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077536.233497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616752640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077536.233510:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077536.233514:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077536.233518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077536.233526:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752768:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077536.233550:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077536.233562:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077536/real 1769077536] req@ffff995b85ac9c00 x1855007616752768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077536.233575:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616752768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077536.233593:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077536.233597:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077536.233601:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077536.233608:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077536.233628:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077536.233640:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077536/real 1769077536] req@ffff995b85acb100 x1855007616752896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077536.233651:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616752896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077536.233661:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077536.233664:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077536.233667:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616752896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077536.233675:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616753024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077536.233697:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077536.233709:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077536/real 1769077536] req@ffff995b85ac8a80 x1855007616753024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077536.233720:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616753024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077536.233729:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077536.233732:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077536.233736:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616753024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077536.233742:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616753152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077536.233763:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077536.233773:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077536/real 1769077536] req@ffff995b85acb800 x1855007616753152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077536.233787:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616753152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077536.233796:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077536.233800:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077536.233803:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616753152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077536.936577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077536.936669:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077537.256580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077537.256611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077537.256626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077537.256723:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077537.960583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077537.960645:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077538.280640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077538.280641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077538.280645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077538.280683:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077538.984573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077538.984630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077539.304583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077539.304583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077539.304583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077539.304585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077540.008572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077540.008635:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077540.328563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077540.328563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077540.328563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077540.328571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077541.032550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077541.032604:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077541.352564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077541.352567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077541.352569:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077541.352569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077541.352580:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077541.352587:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077541.352598:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4527 00000100:00080000:2.0:1769077541.352613:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077541.352623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077541.352623:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077541.352672:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616753280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077541.352705:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077541.352710:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077541.352713:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077541.352718:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4527 00000100:00080000:2.0:1769077541.352723:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077541.352728:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077541.352740:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077541.352744:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616753280:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077541.352760:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616753408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077541.352774:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077541.352778:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077541.352780:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077541.352785:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4527 00000100:00080000:2.0:1769077541.352790:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077541.352795:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077541.352801:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077541.352812:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077541.352817:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616753536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077541.352830:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077541.352835:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077541.352837:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077541.352841:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4527 00000100:00080000:2.0:1769077541.352846:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077541.352849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077541/real 1769077541] req@ffff995b85aca300 x1855007616753280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077541.352854:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077541.352866:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077541.352873:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616753280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077541.352883:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616753664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077541.352896:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077541.352902:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077541.352904:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077541.352909:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077541.352912:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616753280:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077541.352914:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077541.352923:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4527 00000100:00080000:2.0:1769077541.352928:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077541.352931:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077541.352936:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077541.352948:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616753792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077541.352953:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616753408:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077541.352969:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077541.352974:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077541.352977:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077541.352980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077541.352986:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4527 00000100:00080000:2.0:1769077541.352991:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077541.352997:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077541.353002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077541/real 1769077541] req@ffff995b85acaa00 x1855007616753408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077541.353003:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077541.353021:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616753920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077541.353025:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616753408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077541.353045:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077541.353049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077541.353051:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077541.353057:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077541.353060:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077541.353066:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4527 00000100:00080000:2.0:1769077541.353073:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077541.353078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616753408:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077541.353079:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077541.353090:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077541.353100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616753536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077541.353106:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616754048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077541.353128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077541.353140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077541/real 1769077541] req@ffff995b85acb800 x1855007616753536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077541.353149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616753536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077541.353156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077541.353158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077541.353161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616753536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077541.353166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616753664:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077541.353183:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077541.353212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077541/real 1769077541] req@ffff995b85ac8a80 x1855007616753664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077541.353226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616753664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077541.353236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077541.353240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077541.353244:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616753664:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077541.353266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616753792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077541.353285:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077541.353297:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077541/real 1769077541] req@ffff995b85acb100 x1855007616753792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077541.353310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616753792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077541.353319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077541.353323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077541.353327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616753792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077541.353333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616753920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077541.353354:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077541.353367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077541/real 1769077541] req@ffff995b85ac9c00 x1855007616753920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077541.353381:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616753920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077541.353390:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077541.353394:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077541.353397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616753920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077541.353403:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616754048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077541.353419:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077541.353436:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077541/real 1769077541] req@ffff995b85ac9880 x1855007616754048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077541.353447:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616754048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077541.353498:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077541.353502:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077541.353505:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616754048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077542.056589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077542.056686:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077542.376581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077542.376581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077542.376581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077542.376614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077543.080560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077543.080664:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077543.400541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077543.400541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077543.400577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077543.400608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077544.104552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077544.104667:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077544.424523:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077544.424574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077544.424599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077544.424624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077545.128558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077545.128640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077545.448537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077545.448591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077545.448635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077545.448657:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077546.152561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077546.152637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077546.472523:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077546.472552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077546.472567:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077546.472574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077546.472574:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769077546.472575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077546.472578:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077546.472592:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4532 00000100:00080000:3.0:1769077546.472606:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077546.472615:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077546.472667:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616754176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077546.472693:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077546.472696:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077546.472697:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077546.472700:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4532 00000100:00080000:3.0:1769077546.472703:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077546.472706:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077546.472715:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077546.472727:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616754176:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077546.472734:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616754304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077546.472748:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077546.472752:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077546.472754:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077546.472758:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4532 00000100:00080000:3.0:1769077546.472764:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077546.472769:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077546.472774:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077546.472788:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616754432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077546.472795:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077546.472797:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1769077546.472802:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077546.472802:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077546.472810:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4532 00000100:00080000:3.0:1769077546.472815:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077546.472819:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077546.472823:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077546.472844:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077546/real 1769077546] req@ffff995b8455a680 x1855007616754176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077546.472845:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616754560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077546.472881:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077546.472884:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077546.472885:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077546.472890:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616754176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077546.472891:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4532 00000100:00080000:3.0:1769077546.472899:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077546.472904:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077546.472906:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769077546.472910:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077546.472916:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616754176:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769077546.472916:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077546.472937:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616754688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077546.472943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616754304:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077546.472960:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077546.472963:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077546.472966:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077546.472970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077546.472974:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4532 00000100:00080000:3.0:1769077546.472980:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077546.472986:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077546/real 1769077546] req@ffff995b8455b100 x1855007616754304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077546.472986:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077546.472995:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077546.473003:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616754304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077546.473010:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616754816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077546.473013:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077546.473022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077546.473028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616754304:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077546.473041:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077546.473042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616754432:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077546.473048:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077546.473051:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077546.473055:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4532 00000100:00080000:3.0:1769077546.473057:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077546.473060:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077546.473063:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077546.473071:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077546.473075:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616754944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077546.473089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077546/real 1769077546] req@ffff995b8455aa00 x1855007616754432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077546.473123:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616754432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077546.473131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077546.473134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077546.473136:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616754432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077546.473141:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616754560:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077546.473157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077546.473165:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077546/real 1769077546] req@ffff995b84558000 x1855007616754560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077546.473173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616754560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077546.473179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077546.473181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077546.473183:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616754560:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077546.473193:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616754688:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077546.473203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077546.473209:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077546/real 1769077546] req@ffff995b84559880 x1855007616754688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077546.473216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616754688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077546.473227:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077546.473229:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077546.473232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616754688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077546.473235:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616754816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077546.473244:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077546.473250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077546/real 1769077546] req@ffff995b84559c00 x1855007616754816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077546.473257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616754816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077546.473263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077546.473264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077546.473266:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616754816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077546.473269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616754944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077546.473279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077546.473285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077546/real 1769077546] req@ffff995b8455a300 x1855007616754944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077546.473292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616754944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077546.473297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077546.473299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077546.473301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616754944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077547.176551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077547.176611:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077547.496533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077547.496538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077547.496591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077547.496612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077548.200552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077548.200623:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077548.520534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077548.520536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077548.520541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077548.520582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077549.224518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077549.224601:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077549.544562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077549.544563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077549.544579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077549.544587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077550.248560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077550.248620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077550.568525:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077550.568532:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077550.568551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077550.568571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077551.272521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077551.272598:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077551.592558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077551.592558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077551.592562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077551.592569:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077551.592577:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077551.592581:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077551.592589:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4537 00000100:00100000:2.0:1769077551.592590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077551.592602:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077551.592610:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077551.592649:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616755072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077551.592674:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077551.592677:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077551.592679:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077551.592682:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4537 00000100:00080000:3.0:1769077551.592684:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077551.592687:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077551.592695:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077551.592705:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616755200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077551.592713:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077551.592716:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077551.592718:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077551.592720:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616755072:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077551.592723:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4537 00000100:00080000:3.0:1769077551.592730:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077551.592733:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077551.592736:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077551.592743:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616755328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077551.592750:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077551.592752:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077551.592753:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077551.592756:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4537 00000100:00080000:3.0:1769077551.592758:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077551.592760:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077551.592764:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077551.592772:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616755456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077551.592778:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077551.592789:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077551.592791:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077551.592794:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077551.592798:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4537 00000100:00080000:3.0:1769077551.592802:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077551.592805:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077551.592809:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077551.592818:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616755584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077551.592825:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077551.592829:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769077551.592830:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077551/real 1769077551] req@ffff995b84558700 x1855007616755072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077551.592831:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077551.592838:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4537 00000100:00080000:3.0:1769077551.592844:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077551.592848:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077551.592852:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077551.592857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616755072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077551.592862:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616755712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077551.592877:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077551.592877:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077551.592884:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077551.592886:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077551.592887:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077551.592892:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616755072:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077551.592894:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4537 00000100:00080000:3.0:1769077551.592902:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077551.592905:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077551.592907:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077551.592914:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616755840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077551.592922:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616755200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077551.592940:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077551.592949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077551/real 1769077551] req@ffff995b8455b480 x1855007616755200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077551.592957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616755200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077551.592963:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077551.592966:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077551.592968:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616755200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077551.592971:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616755328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077551.592982:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077551.592988:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077551/real 1769077551] req@ffff995b8455ad80 x1855007616755328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077551.592995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616755328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077551.593002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077551.593004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077551.593005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616755328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077551.593009:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616755456:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077551.593041:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077551.593050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077551/real 1769077551] req@ffff995b8455a300 x1855007616755456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077551.593058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616755456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077551.593064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077551.593066:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077551.593069:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616755456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077551.593073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616755584:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077551.593085:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077551.593091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077551/real 1769077551] req@ffff995b84559c00 x1855007616755584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077551.593099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616755584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077551.593105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077551.593107:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077551.593108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616755584:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077551.593112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616755712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077551.593121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077551.593132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077551/real 1769077551] req@ffff995b84559880 x1855007616755712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077551.593141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616755712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077551.593146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077551.593148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077551.593150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616755712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077551.593162:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616755840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077551.593173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077551.593180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077551/real 1769077551] req@ffff995b84558000 x1855007616755840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077551.593187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616755840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077551.593193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077551.593195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077551.593196:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616755840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077552.296600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077552.296684:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077552.616546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077552.616546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077552.616560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077552.616595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077553.320533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077553.320598:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077553.640562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077553.640562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077553.640562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077553.640594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077554.344571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077554.344664:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077554.664571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077554.664571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077554.664571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077554.664571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077555.368529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077555.368595:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077555.688523:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077555.688538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077555.688557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077555.688564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077556.392556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077556.392603:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077556.712563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077556.712587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077556.712587:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077556.712595:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077556.712599:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077556.712605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077556.712609:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4542 00000100:00080000:3.0:1769077556.712622:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077556.712627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077556.712629:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077556.712669:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616755968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077556.712692:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077556.712695:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077556.712696:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077556.712700:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4542 00000100:00080000:3.0:1769077556.712703:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077556.712706:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077556.712713:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077556.712729:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616755968:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077556.712731:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616756096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077556.712746:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077556.712750:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077556.712753:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077556.712757:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4542 00000100:00080000:3.0:1769077556.712762:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077556.712767:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077556.712773:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077556.712787:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616756224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077556.712797:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077556.712803:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077556.712807:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077556.712809:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077556.712813:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4542 00000100:00080000:3.0:1769077556.712817:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077556.712819:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077556.712824:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077556.712835:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616756352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077556.712838:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077556/real 1769077556] req@ffff995b8455aa00 x1855007616755968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077556.712852:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077556.712857:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077556.712859:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077556.712863:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4542 00000100:00100000:1.0:1769077556.712866:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616755968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077556.712867:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077556.712874:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077556.712880:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769077556.712885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077556.712890:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616756480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077556.712893:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077556.712901:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616755968:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077556.712905:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077556.712909:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077556.712910:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077556.712914:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4542 00000100:00080000:3.0:1769077556.712916:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077556.712918:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077556.712921:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077556.712943:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616756608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077556.712955:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077556.712959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756096:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077556.712960:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077556.712964:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077556.712970:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4542 00000100:00080000:3.0:1769077556.712974:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077556.712977:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077556.712982:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077556.712991:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616756736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077556.712998:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077556.713017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077556/real 1769077556] req@ffff995b8455b100 x1855007616756096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077556.713027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616756096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077556.713033:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077556.713035:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077556.713038:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077556.713042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077556.713055:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077556.713061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077556/real 1769077556] req@ffff995b8455a680 x1855007616756224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077556.713068:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616756224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077556.713075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077556.713077:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077556.713079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077556.713083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756352:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077556.713093:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077556.713099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077556/real 1769077556] req@ffff995b84558a80 x1855007616756352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077556.713106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616756352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077556.713112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077556.713114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077556.713116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077556.713119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756480:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077556.713127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077556.713133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077556/real 1769077556] req@ffff995b8455bb80 x1855007616756480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077556.713140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616756480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077556.713146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077556.713148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077556.713150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077556.713161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616756608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077556.713170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077556.713177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077556/real 1769077556] req@ffff995b84559500 x1855007616756608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077556.713184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616756608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077556.713193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077556.713195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077556.713197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616756608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077556.713201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616756736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077556.713209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077556.713215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077556/real 1769077556] req@ffff995b8455aa00 x1855007616756736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077556.713222:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616756736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077556.713227:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077556.713229:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077556.713231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616756736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077557.416537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077557.416593:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077557.736551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077557.736557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077557.736564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077557.736581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077558.440539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077558.440605:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077558.760531:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077558.760531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077558.760545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077558.760653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077559.464557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077559.464650:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077559.784585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077559.784585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077559.784619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077559.784623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077560.488522:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077560.488621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077560.808610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077560.808610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077560.808625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077560.808630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077561.512585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077561.512655:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077561.832562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077561.832574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077561.832629:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077561.832638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077561.832642:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077561.832650:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077561.832656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077561.832663:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4547 00000100:00080000:3.0:1769077561.832679:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077561.832689:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077561.832753:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616756864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077561.832794:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077561.832799:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077561.832803:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077561.832811:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4547 00000100:00080000:3.0:1769077561.832817:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077561.832823:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077561.832836:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077561.832840:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616756864:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077561.832891:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616756992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077561.832902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077561.832905:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077561.832908:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077561.832911:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4547 00000100:00080000:3.0:1769077561.832915:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077561.832919:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077561.832923:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077561.832932:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616757120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077561.832938:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077561.832941:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077561.832942:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077561.832952:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4547 00000800:00000100:1.0:1769077561.832953:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077561.832961:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077561.832966:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077561.832970:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077561.832984:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616757248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077561.832993:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077561.832995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077561.832998:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077561.833003:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4547 00000100:00080000:3.0:1769077561.833008:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077561.833012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077561/real 1769077561] req@ffff995b84558000 x1855007616756864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077561.833018:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077561.833034:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077561.833041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616756864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077561.833053:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077561.833055:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616757376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077561.833061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077561.833071:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616756864:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077561.833090:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077561.833095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077561.833097:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077561.833109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756992:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077561.833111:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4547 00000100:00080000:3.0:1769077561.833126:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077561.833135:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769077561.833142:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769077561.833151:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077561.833157:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077561/real 1769077561] req@ffff995b84559880 x1855007616756992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077561.833168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616756992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077561.833178:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077561.833194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077561.833194:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616757504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077561.833198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616756992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077561.833203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616757120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077561.833217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077561.833229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077561/real 1769077561] req@ffff995b84559c00 x1855007616757120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077561.833241:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077561.833244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616757120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077561.833259:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077561.833269:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077561.833275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077561.833282:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4547 00000100:00080000:1.0:1769077561.833283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077561.833288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616757120:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077561.833296:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077561.833297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616757248:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077561.833310:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769077561.833317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769077561.833323:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077561.833334:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077561/real 1769077561] req@ffff995b8455a300 x1855007616757248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077561.833347:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616757632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077561.833354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616757248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077561.833399:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077561.833405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077561.833408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616757248:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077561.833414:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616757376:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077561.833436:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077561.833445:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077561/real 1769077561] req@ffff995b8455ad80 x1855007616757376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077561.833473:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616757376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077561.833480:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077561.833483:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077561.833485:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616757376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077561.833502:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616757504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077561.833515:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077561.833522:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077561/real 1769077561] req@ffff995b8455b480 x1855007616757504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077561.833530:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616757504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077561.833536:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077561.833538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077561.833541:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616757504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077561.833545:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616757632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077561.833555:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077561.833561:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077561/real 1769077561] req@ffff995b84558700 x1855007616757632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077561.833579:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616757632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077561.833584:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077561.833586:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077561.833588:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616757632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077562.536563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077562.536637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077562.856575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077562.856575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077562.856585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077562.856681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077563.560554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077563.560633:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077563.880544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077563.880544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077563.880544:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077563.880553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077564.584575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077564.584649:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077564.904558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077564.904558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077564.904558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077564.904573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077565.608562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077565.608632:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077565.928553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077565.928555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077565.928563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077565.928581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077566.632541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077566.632586:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077566.952552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077566.952578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077566.952616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077566.952618:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077566.952625:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077566.952628:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077566.952637:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4552 00000100:00080000:1.0:1769077566.952649:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077566.952657:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077566.952661:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077566.952696:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616757760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077566.952724:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077566.952728:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077566.952730:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077566.952738:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4552 00000100:00080000:1.0:1769077566.952741:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077566.952748:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077566.952771:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077566.952771:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616757760:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077566.952791:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616757888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077566.952807:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077566.952811:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077566.952814:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077566.952818:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4552 00000100:00080000:1.0:1769077566.952824:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077566.952828:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077566.952834:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077566.952839:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077566.952863:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616758016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077566.952873:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077566.952876:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077566.952878:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077566.952881:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4552 00000100:00080000:1.0:1769077566.952884:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077566.952887:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077566.952892:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077566.952904:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616758144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077566.952910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077566.952912:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077566.952914:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077566.952917:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4552 00000100:00080000:1.0:1769077566.952918:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077566.952920:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077566.952923:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077566.952929:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616758272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077566.952935:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077566.952938:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077566.952940:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077566.952943:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4552 00000100:00080000:1.0:1769077566.952946:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077566.952949:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077566.952953:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077566.952962:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077566/real 1769077566] req@ffff995b868c7100 x1855007616757760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077566.952966:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616758400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077566.952998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077566.953002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077566.953005:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077566.953007:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616757760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077566.953012:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4552 00000100:00080000:1.0:1769077566.953018:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077566.953024:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077566.953025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769077566.953028:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077566.953036:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077566.953041:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616758528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077566.953047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616757760:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077566.953183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616757888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077566.953245:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077566.953272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077566/real 1769077566] req@ffff995b868c6a00 x1855007616757888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077566.953284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616757888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077566.953291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077566.953294:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077566.953297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616757888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077566.953301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077566.953314:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077566.953321:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077566/real 1769077566] req@ffff995b868c5c00 x1855007616758016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077566.953333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616758016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077566.953348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077566.953350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077566.953352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077566.953356:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758144:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077566.953369:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077566.953375:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077566/real 1769077566] req@ffff995b868c7480 x1855007616758144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077566.953386:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616758144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077566.953395:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077566.953398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077566.953401:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758144:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077566.953408:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077566.953428:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077566.953440:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077566/real 1769077566] req@ffff995b868c7800 x1855007616758272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077566.953452:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616758272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077566.953507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077566.953511:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077566.953514:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077566.953521:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616758400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077566.953541:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077566.953550:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077566/real 1769077566] req@ffff995b868c5500 x1855007616758400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077566.953557:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616758400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077566.953563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077566.953565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077566.953567:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616758400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077566.953570:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616758528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077566.953580:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077566.953586:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077566/real 1769077566] req@ffff995b868c4700 x1855007616758528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077566.953594:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616758528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077566.953599:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077566.953601:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077566.953603:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616758528:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769077567.656598:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077567.656657:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077567.976555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077567.976611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077567.976627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077567.976657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077568.680543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077568.680665:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077569.000629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077569.000632:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077569.000634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077569.000635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077569.704613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077569.704767:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077570.024526:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077570.024534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077570.024569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077570.024596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077570.728561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077570.728636:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077571.048573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077571.048573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077571.048604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077571.048604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077571.752612:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077571.752717:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077572.072615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077572.072645:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077572.072654:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077572.072740:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077572.072743:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077572.072754:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077572.072759:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077572.072770:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4557 00000100:00080000:3.0:1769077572.072782:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077572.072789:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077572.072827:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616758656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077572.072854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077572.072874:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077572.072875:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077572.072879:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4557 00000100:00080000:3.0:1769077572.072882:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077572.072886:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077572.072895:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077572.072905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616758656:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077572.072908:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616758784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077572.072918:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077572.072921:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077572.072922:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077572.072924:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4557 00000100:00080000:3.0:1769077572.072926:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077572.072929:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077572.072932:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077572.072940:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616758912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077572.072947:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077572.072949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077572.072950:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077572.072952:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4557 00000100:00080000:3.0:1769077572.072954:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077572.072956:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077572.072959:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077572.072966:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616759040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077572.072972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077572.072974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077572.072975:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077572.072981:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077572.072981:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4557 00000100:00080000:3.0:1769077572.072987:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077572.072990:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077572.072994:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077572.073002:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616759168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077572.073009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077572.073012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077572.073013:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077572.073016:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4557 00000100:00080000:3.0:1769077572.073018:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077572.073021:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077572.073024:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077572.073032:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616759296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077572.073038:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077572/real 1769077572] req@ffff995b84558700 x1855007616758656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077572.073043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077572.073051:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077572.073055:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077572.073060:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4557 00000100:00080000:3.0:1769077572.073063:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077572.073066:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077572.073087:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616758656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077572.073089:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769077572.073103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077572.073103:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616759424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077572.073112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077572.073121:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616758656:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077572.073144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077572.073165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077572.073174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077572/real 1769077572] req@ffff995b8455b480 x1855007616758784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077572.073183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616758784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077572.073189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077572.073192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077572.073194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077572.073198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077572.073209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077572.073216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077572/real 1769077572] req@ffff995b8455ad80 x1855007616758912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077572.073224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616758912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077572.073231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077572.073233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077572.073235:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616758912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077572.073239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759040:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077572.073250:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077572.073257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077572/real 1769077572] req@ffff995b8455a300 x1855007616759040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077572.073264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616759040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077572.073270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077572.073272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077572.073274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077572.073278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759168:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077572.073287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077572.073293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077572/real 1769077572] req@ffff995b84559c00 x1855007616759168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077572.073306:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616759168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077572.073313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077572.073314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077572.073316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077572.073320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616759296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077572.073331:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077572.073337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077572/real 1769077572] req@ffff995b84559880 x1855007616759296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077572.073345:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616759296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077572.073367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077572.073370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077572.073373:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616759296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077572.073379:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616759424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077572.073398:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077572.073410:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077572/real 1769077572] req@ffff995b84558000 x1855007616759424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077572.073424:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616759424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077572.073437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077572.073440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077572.073442:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616759424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077572.776518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077572.776652:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077573.096590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077573.096604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077573.096626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077573.096635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077573.800544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077573.800641:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077574.120609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077574.120651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077574.120677:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077574.120709:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077574.824520:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077574.824594:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077575.144565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077575.144565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077575.144566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077575.144586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077575.848581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077575.848644:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077576.168510:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077576.168532:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077576.168536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077576.168554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077576.872548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077576.872625:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077577.192550:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077577.192550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077577.192559:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077577.192565:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077577.192568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077577.192568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077577.192577:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4562 00000100:00080000:0.0:1769077577.192603:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077577.192608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077577.192612:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077577.192655:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616759552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077577.192683:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077577.192688:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077577.192690:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077577.192695:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4562 00000100:00080000:0.0:1769077577.192699:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077577.192703:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077577.192714:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077577.192726:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616759680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077577.192736:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077577.192739:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077577.192740:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077577.192743:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4562 00000100:00080000:0.0:1769077577.192745:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077577.192748:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077577.192752:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077577.192758:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616759808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077577.192764:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077577.192767:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077577.192769:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077577.192771:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4562 00000100:00080000:0.0:1769077577.192773:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077577.192775:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077577.192779:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077577.192785:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616759936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077577.192792:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616759552:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077577.192794:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077577.192801:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077577.192804:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077577.192807:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4562 00000100:00080000:0.0:1769077577.192809:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077577.192811:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077577.192814:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077577.192822:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616760064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077577.192828:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077577.192830:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077577.192832:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077577.192835:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4562 00000100:00080000:0.0:1769077577.192837:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077577.192840:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077577.192842:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077577.192849:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616760192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077577.192854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077577.192856:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077577.192857:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077577.192860:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4562 00000100:00080000:0.0:1769077577.192863:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077577.192865:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077577.192868:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077577.192875:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077577.192877:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616760320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077577.192925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077577/real 1769077577] req@ffff995b84641500 x1855007616759552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077577.192948:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616759552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077577.192960:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077577.192965:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077577.192969:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616759552:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077577.192978:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077577.193001:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077577.193014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077577/real 1769077577] req@ffff995b84641180 x1855007616759680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077577.193027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616759680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077577.193036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077577.193040:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077577.193043:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077577.193050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077577.193068:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077577.193079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077577/real 1769077577] req@ffff995b84643800 x1855007616759808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077577.193090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616759808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077577.193101:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077577.193104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077577.193108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077577.193136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759936:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077577.193155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077577.193166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077577/real 1769077577] req@ffff995b84643480 x1855007616759936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077577.193177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616759936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077577.193187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077577.193190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077577.193194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616759936:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077577.193200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760064:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077577.193217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077577.193228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077577/real 1769077577] req@ffff995b84642d80 x1855007616760064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077577.193240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616760064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077577.193249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077577.193252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077577.193255:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760064:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077577.193261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616760192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077577.193279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077577.193290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077577/real 1769077577] req@ffff995b84643100 x1855007616760192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077577.193302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616760192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077577.193311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077577.193314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077577.193317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616760192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077577.193324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616760320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077577.193343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077577.193354:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077577/real 1769077577] req@ffff995b84643b80 x1855007616760320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077577.193366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616760320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077577.193376:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077577.193379:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077577.193382:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616760320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077577.896557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077577.896624:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077578.216550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077578.216550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077578.216552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077578.216600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077578.920571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077578.920627:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077579.240573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077579.240579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077579.240640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077579.240655:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077579.944593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077579.944671:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077580.264582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077580.264582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077580.264609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077580.264740:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077580.968604:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077580.968664:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077581.288569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077581.288569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077581.288572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077581.288626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077581.992565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077581.992622:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077582.312569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077582.312575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077582.312575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077582.312584:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077582.312592:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077582.312598:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077582.312602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077582.312609:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4568 00000100:00080000:0.0:1769077582.312620:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077582.312627:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077582.312664:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616760448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077582.312687:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077582.312690:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077582.312691:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077582.312694:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4568 00000100:00080000:0.0:1769077582.312697:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077582.312700:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077582.312708:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077582.312718:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616760448:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077582.312722:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616760576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077582.312731:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077582.312733:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077582.312735:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077582.312737:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4568 00000100:00080000:0.0:1769077582.312741:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077582.312743:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077582.312746:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077582.312753:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616760704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077582.312760:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077582.312763:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077582.312765:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077582.312767:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4568 00000100:00080000:0.0:1769077582.312779:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077582.312781:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077582.312785:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769077582.312792:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077582.312795:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616760832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077582.312803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077582.312805:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077582.312807:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077582.312809:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4568 00000100:00080000:0.0:1769077582.312814:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077582.312816:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077582.312819:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077582.312826:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616760960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077582.312833:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077582.312837:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769077582.312837:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077582/real 1769077582] req@ffff995b84642a00 x1855007616760448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077582.312841:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077582.312849:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4568 00000100:00080000:0.0:1769077582.312853:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077582.312860:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077582.312866:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077582.312875:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616760448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077582.312876:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616761088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077582.312902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077582.312903:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077582.312908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077582.312911:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077582.312912:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077582.312918:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4568 00000100:00100000:1.0:1769077582.312921:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616760448:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077582.312923:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077582.312930:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077582.312934:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077582.312941:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616761216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077582.312951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077582.312972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077582.312981:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077582/real 1769077582] req@ffff995b84640700 x1855007616760576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077582.312990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616760576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077582.312996:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077582.312998:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077582.313000:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077582.313005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077582.313024:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077582.313035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077582/real 1769077582] req@ffff995b84641880 x1855007616760704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077582.313048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616760704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077582.313066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077582.313069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077582.313073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077582.313080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760832:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077582.313099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077582.313109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077582/real 1769077582] req@ffff995b84643b80 x1855007616760832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077582.313117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616760832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077582.313125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077582.313129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077582.313133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077582.313139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760960:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077582.313157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077582.313167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077582/real 1769077582] req@ffff995b84643100 x1855007616760960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077582.313181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616760960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077582.313190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077582.313194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077582.313198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616760960:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077582.313204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616761088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077582.313224:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077582.313236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077582/real 1769077582] req@ffff995b84642d80 x1855007616761088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077582.313248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616761088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077582.313257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077582.313260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077582.313264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616761088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077582.313283:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616761216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077582.313313:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077582.313322:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077582/real 1769077582] req@ffff995b84643480 x1855007616761216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077582.313330:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616761216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077582.313335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077582.313337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077582.313339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616761216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077583.016581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077583.016643:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077583.336552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077583.336558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077583.336577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077583.336592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077584.040574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077584.040637:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077584.360574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077584.360574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077584.360574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077584.360600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077585.064563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077585.064651:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077585.384535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077585.384538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077585.384539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077585.384542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077586.088524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077586.088597:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077586.408568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077586.408568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077586.408572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077586.408595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077587.112566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077587.112632:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077587.432530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077587.432534:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077587.432563:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077587.432571:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077587.432578:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077587.432584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077587.432590:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4573 00000100:00080000:0.0:1769077587.432602:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077587.432604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077587.432611:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077587.432654:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616761344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077587.432683:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077587.432687:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077587.432689:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077587.432694:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4573 00000100:00080000:0.0:1769077587.432698:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077587.432702:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077587.432712:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077587.432717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616761344:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077587.432735:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616761472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077587.432746:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077587.432749:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077587.432751:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077587.432754:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4573 00000100:00080000:0.0:1769077587.432770:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077587.432774:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077587.432778:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077587.432780:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077587.432788:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616761600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077587.432797:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077587.432801:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077587.432802:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077587.432806:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4573 00000100:00080000:0.0:1769077587.432810:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077587.432814:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077587.432818:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077587.432818:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077587/real 1769077587] req@ffff995b84643800 x1855007616761344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077587.432827:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616761728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077587.432836:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077587.432836:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616761344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077587.432839:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077587.432842:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077587.432845:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4573 00000100:00080000:0.0:1769077587.432848:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077587.432848:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077587.432851:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769077587.432852:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769077587.432856:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077587.432856:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616761344:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077587.432865:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616761856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077587.432872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077587.432875:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077587.432877:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077587.432879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616761472:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077587.432881:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4573 00000100:00080000:0.0:1769077587.432884:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077587.432888:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077587.432892:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077587.432900:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077587.432901:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616761984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077587.432912:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077587/real 1769077587] req@ffff995b84641180 x1855007616761472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077587.432918:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077587.432921:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077587.432924:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077587.432925:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616761472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077587.432928:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4573 00000100:00080000:0.0:1769077587.432932:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077587.432934:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077587.432936:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769077587.432937:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769077587.432941:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077587.432942:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616761472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077587.432948:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616761600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077587.432950:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616762112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077587.432964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077587.432975:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077587/real 1769077587] req@ffff995b84641500 x1855007616761600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077587.432988:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616761600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077587.432998:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077587.433001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077587.433003:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616761600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077587.433008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616761728:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077587.433023:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077587.433030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077587/real 1769077587] req@ffff995b84642680 x1855007616761728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077587.433038:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616761728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077587.433044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077587.433046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077587.433048:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616761728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077587.433051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616761856:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077587.433060:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077587.433067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077587/real 1769077587] req@ffff995b84640e00 x1855007616761856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077587.433074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616761856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077587.433080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077587.433082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077587.433084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616761856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077587.433097:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616761984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077587.433107:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077587.433113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077587/real 1769077587] req@ffff995b84641c00 x1855007616761984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077587.433120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616761984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077587.433125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077587.433127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077587.433129:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616761984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077587.433132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616762112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077587.433141:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077587.433147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077587/real 1769077587] req@ffff995b84640a80 x1855007616762112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077587.433154:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616762112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077587.433159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077587.433162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077587.433164:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616762112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077588.136540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077588.136580:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077588.456583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077588.456584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077588.456584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077588.456622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077589.160530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077589.160587:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077589.480542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077589.480542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077589.480547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077589.480558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077590.184511:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077590.184584:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077590.504561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077590.504561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077590.504561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077590.504573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077591.208538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077591.208585:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077591.528584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077591.528585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077591.528585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077591.528586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077592.232523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077592.232604:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077592.552550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077592.552552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077592.552575:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077592.552582:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077592.552587:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077592.552596:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4578 00000100:00080000:3.0:1769077592.552610:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077592.552616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077592.552617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077592.552619:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077592.552672:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616762240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077592.552702:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077592.552706:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077592.552708:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077592.552711:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4578 00000100:00080000:3.0:1769077592.552715:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077592.552719:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077592.552728:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077592.552743:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616762368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077592.552745:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616762240:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077592.552760:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077592.552763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077592.552764:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077592.552767:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4578 00000100:00080000:3.0:1769077592.552769:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077592.552772:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077592.552775:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077592.552783:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616762496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077592.552790:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077592.552792:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077592.552793:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077592.552796:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4578 00000100:00080000:3.0:1769077592.552798:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077592.552800:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077592.552803:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077592.552812:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616762624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077592.552817:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077592.552824:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077592.552827:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077592.552828:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077592.552832:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4578 00000100:00080000:3.0:1769077592.552851:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077592.552855:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077592.552858:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077592.552867:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616762752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077592.552873:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077592.552877:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077592/real 1769077592] req@ffff995b8455aa00 x1855007616762240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077592.552878:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077592.552881:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077592.552889:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4578 00000100:00080000:3.0:1769077592.552894:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077592.552898:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077592.552902:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077592.552923:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616762240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077592.552930:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616762880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077592.552941:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077592.552946:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077592.552951:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077592.552955:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077592.552959:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077592.552960:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616762240:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077592.552969:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4578 00000100:00080000:3.0:1769077592.552974:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077592.552988:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077592.552992:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616762368:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769077592.552998:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077592.553012:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616763008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077592.553024:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077592.553042:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077592/real 1769077592] req@ffff995b84559500 x1855007616762368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077592.553058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616762368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077592.553069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077592.553071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077592.553074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616762368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077592.553078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616762496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077592.553094:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077592.553102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077592/real 1769077592] req@ffff995b8455bb80 x1855007616762496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077592.553110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616762496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077592.553117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077592.553119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077592.553121:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616762496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077592.553125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616762624:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077592.553136:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077592.553141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077592/real 1769077592] req@ffff995b84558a80 x1855007616762624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077592.553149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616762624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077592.553159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077592.553161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077592.553163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616762624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077592.553166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616762752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077592.553191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077592.553198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077592/real 1769077592] req@ffff995b8455a680 x1855007616762752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077592.553205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616762752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077592.553212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077592.553214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077592.553216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616762752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077592.553219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616762880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077592.553230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077592.553236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077592/real 1769077592] req@ffff995b8455b100 x1855007616762880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077592.553244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616762880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077592.553249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077592.553251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077592.553253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616762880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077592.553267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616763008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077592.553277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077592.553284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077592/real 1769077592] req@ffff995b8455aa00 x1855007616763008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077592.553291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616763008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077592.553297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077592.553299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077592.553301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616763008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077593.256618:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077593.256697:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077593.576586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077593.576586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077593.576665:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077593.576923:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077594.280535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077594.280606:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077594.600575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077594.600575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077594.600583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077594.600589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077595.304554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077595.304626:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077595.624562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077595.624562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077595.624566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077595.624609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077596.328529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077596.328596:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077596.648556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077596.648556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077596.648557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077596.648605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077597.352545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077597.352634:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077597.672583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077597.672583:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077597.672584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077597.672590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077597.672595:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077597.672603:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077597.672614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077597.672615:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4583 00000100:00080000:2.0:1769077597.672628:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077597.672635:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077597.672672:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616763136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077597.672696:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077597.672699:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077597.672700:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077597.672703:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4583 00000100:00080000:2.0:1769077597.672706:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077597.672710:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077597.672718:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077597.672732:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616763136:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077597.672732:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616763264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077597.672746:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077597.672748:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077597.672749:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077597.672752:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4583 00000100:00080000:2.0:1769077597.672754:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077597.672757:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077597.672760:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077597.672770:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616763392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077597.672777:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077597.672779:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077597.672780:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077597.672782:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4583 00000100:00080000:2.0:1769077597.672784:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077597.672787:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077597.672790:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077597.672798:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616763520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077597.672804:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077597.672806:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077597.672808:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077597.672810:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4583 00000100:00080000:2.0:1769077597.672812:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077597.672814:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077597.672818:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077597.672826:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616763648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077597.672833:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077597.672835:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077597.672837:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077597.672840:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616763264:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077597.672843:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4583 00000100:00080000:2.0:1769077597.672850:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077597.672853:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077597.672857:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077597.672867:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616763776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077597.672874:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077597.672877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077597.672879:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077597.672881:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4583 00000100:00080000:2.0:1769077597.672884:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077597.672887:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077597.672891:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616763392:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077597.672891:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077597.672912:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616763904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077597.672922:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616763520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077597.672957:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616763648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077597.672984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616763776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077597.673005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616763904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077598.376572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077598.376629:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077598.696584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077598.696603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077598.696640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077598.696668:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077599.400559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077599.400616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077599.720561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077599.720561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077599.720588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077599.720603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077600.424571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077600.424640:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077600.744573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077600.744573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077600.744573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077600.744632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:1.0:1769077600.744783:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769077600.744826:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077600.744847:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077600.744912:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077600.744922:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077600.744931:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077600.744939:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077600.744947:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077600.744956:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769077600.745093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077597/real 1769077600] req@ffff995b85ac8a80 x1855007616763776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077600.745122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616763776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077600.745135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077600.745141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077600.745146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616763776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077600.745154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077597/real 1769077600] req@ffff995b85acb800 x1855007616763904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077600.745169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616763904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077600.745178:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077600.745184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077600.745202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616763904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077600.745211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077597/real 1769077600] req@ffff995b85acbb80 x1855007616763264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077600.745226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616763264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077600.745236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077600.745242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077600.745245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616763264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077600.745255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077597/real 1769077600] req@ffff995b85ac9880 x1855007616763392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077600.745280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616763392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077600.745292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077600.745298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077600.745303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616763392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077600.745312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077597/real 1769077600] req@ffff995b85ac9c00 x1855007616763520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077600.745324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616763520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077600.745335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077600.745340:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077600.745344:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616763520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077600.745352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077597/real 1769077600] req@ffff995b85acb100 x1855007616763648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077600.745368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616763648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077600.745380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077600.745384:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077600.745388:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616763648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077600.745396:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077597/real 1769077600] req@ffff995b85ac8380 x1855007616763136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077600.745411:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616763136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077600.745422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077600.745425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077600.745430:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616763136:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077601.448554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077601.448618:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077601.768567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077601.768583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077601.768631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077601.768674:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077602.472569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077602.472635:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077602.792560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077602.792560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077602.792585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077602.792602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077602.792735:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077602.792743:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077602.792748:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077602.792759:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4588 00000100:00080000:0.0:1769077602.792775:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077602.792783:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077602.792825:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616764032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077602.792860:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077602.792864:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077602.792867:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077602.792871:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4588 00000100:00080000:0.0:1769077602.792876:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077602.792880:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077602.792892:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077602.792905:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616764160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077602.792914:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077602.792918:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077602.792920:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077602.792925:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4588 00000100:00080000:0.0:1769077602.792928:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077602.792932:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077602.792938:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077602.792948:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616764288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077602.792957:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077602.792961:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077602.792963:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077602.792967:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4588 00000100:00080000:0.0:1769077602.792970:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077602.792974:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077602.792980:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077602.792992:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616764416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077602.793006:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616764032:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077602.793007:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077602.793015:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077602.793019:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077602.793025:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4588 00000100:00080000:0.0:1769077602.793028:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077602.793032:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077602.793038:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077602.793050:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616764544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077602.793061:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077602.793065:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077602.793067:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077602.793072:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4588 00000100:00080000:0.0:1769077602.793076:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077602.793080:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077602.793086:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077602.793090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077602.793102:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616764672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077602.793113:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077602.793117:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077602.793119:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077602.793124:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4588 00000100:00080000:0.0:1769077602.793128:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077602.793134:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077602.793141:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077602.793141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077602/real 1769077602] req@ffff995b84640000 x1855007616764032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077602.793177:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616764800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077602.793198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616764032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.793212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.793219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077602.793224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616764032:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077602.793233:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616764160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077602.793260:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077602.793274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077602/real 1769077602] req@ffff995b84641f80 x1855007616764160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077602.793289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616764160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.793298:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.793302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077602.793308:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616764160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077602.793315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616764288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077602.793727:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077602.793743:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077602/real 1769077602] req@ffff995b84640380 x1855007616764288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077602.793758:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616764288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.793779:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.793784:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077602.793789:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616764288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077602.793829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616764416:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077602.793855:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077602.793871:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077602/real 1769077602] req@ffff995b84640a80 x1855007616764416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077602.793887:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616764416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.793898:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.793903:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077602.793907:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616764416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077602.793915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616764544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077602.793936:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077602.793949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077602/real 1769077602] req@ffff995b84641c00 x1855007616764544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077602.793962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616764544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.793973:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.793976:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077602.793981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616764544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077602.793987:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616764672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077602.794008:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077602.794020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077602/real 1769077602] req@ffff995b84640e00 x1855007616764672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077602.794032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616764672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.794040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.794043:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077602.794046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616764672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077602.794051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616764800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077602.794069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077602.794080:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077602/real 1769077602] req@ffff995b84642680 x1855007616764800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077602.794091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616764800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.794099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.794102:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077602.794104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616764800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077603.496548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077603.496616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077603.816550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077603.816553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077603.816585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077603.816665:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077604.520551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077604.520605:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077604.840526:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077604.840529:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077604.840560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077604.840561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077605.544549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077605.544621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077605.864578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077605.864578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077605.864595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077605.864622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077606.568573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077606.568639:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077606.888586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077606.888597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077606.891051:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077606.891069:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077607.592550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077607.592623:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077607.912573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077607.912573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077607.912573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077607.912574:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077607.912585:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077607.912590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077607.912602:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4593 00000100:00080000:0.0:1769077607.912613:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077607.912622:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077607.912627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077607.912672:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616764928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077607.912699:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077607.912703:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077607.912706:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077607.912710:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4593 00000100:00080000:0.0:1769077607.912715:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077607.912720:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077607.912730:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077607.912730:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616764928:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077607.912743:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616765056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077607.912753:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077607.912757:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077607.912758:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077607.912762:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4593 00000100:00080000:0.0:1769077607.912765:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077607.912768:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077607.912773:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077607.912781:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616765184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077607.912790:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077607.912793:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077607.912794:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077607.912795:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077607.912798:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4593 00000100:00080000:0.0:1769077607.912801:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077607.912805:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077607.912809:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077607.912817:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616765312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077607.912826:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077607.912829:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077607.912830:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077607.912833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077607/real 1769077607] req@ffff995b84641500 x1855007616764928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077607.912834:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4593 00000100:00080000:0.0:1769077607.912838:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077607.912841:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077607.912846:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077607.912852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616764928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077607.912853:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616765440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077607.912863:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077607.912864:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077607.912867:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077607.912869:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077607.912869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077607.912873:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4593 00000100:00100000:3.0:1769077607.912873:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616764928:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077607.912877:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077607.912880:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077607.912884:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077607.912892:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616765568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077607.912897:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765056:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077607.912902:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077607.912906:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077607.912908:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077607.912911:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4593 00000100:00080000:0.0:1769077607.912915:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077607.912917:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077607.912919:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077607.912924:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077607.912929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077607/real 1769077607] req@ffff995b84641180 x1855007616765056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077607.912933:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616765696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077607.912942:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616765056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077607.912951:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077607.912955:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077607.912958:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077607.912964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077607.912987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077607.912999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077607/real 1769077607] req@ffff995b84643800 x1855007616765184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077607.913011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616765184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077607.913019:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077607.913021:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077607.913024:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077607.913028:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765312:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077607.913043:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077607.913050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077607/real 1769077607] req@ffff995b84643480 x1855007616765312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077607.913058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616765312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077607.913064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077607.913066:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077607.913067:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077607.913086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077607.913100:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077607.913107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077607/real 1769077607] req@ffff995b84642d80 x1855007616765440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077607.913120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616765440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077607.913127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077607.913129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077607.913131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077607.913145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616765568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077607.913156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077607.913162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077607/real 1769077607] req@ffff995b84643100 x1855007616765568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077607.913170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616765568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077607.913175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077607.913178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077607.913180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616765568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077607.913183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616765696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077607.913192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077607.913197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077607/real 1769077607] req@ffff995b84643b80 x1855007616765696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077607.913204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616765696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077607.913214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077607.913217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077607.913219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616765696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077608.616559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077608.616614:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077608.936516:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077608.936550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077608.936563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077608.936583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077609.640525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077609.640619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077609.960684:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077609.960857:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077609.960890:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077609.961063:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077610.664610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077610.664708:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077610.984542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077610.984562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077610.985510:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077610.985523:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077611.688519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077611.688572:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077612.008561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077612.008640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077612.008681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077612.008741:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077612.712566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077612.712633:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077613.032518:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077613.032547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077613.032549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077613.032565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077613.032580:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077613.032588:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077613.032593:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077613.032604:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4598 00000100:00080000:0.0:1769077613.032616:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077613.032623:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077613.032675:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616765824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077613.032707:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077613.032710:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077613.032712:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077613.032716:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4598 00000100:00080000:0.0:1769077613.032720:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077613.032725:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077613.032735:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077613.032750:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616765824:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077613.032752:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616765952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077613.032768:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077613.032771:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077613.032773:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077613.032777:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4598 00000100:00080000:0.0:1769077613.032782:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077613.032787:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077613.032792:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077613.032805:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616766080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077613.032815:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077613.032819:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077613.032821:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077613.032825:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4598 00000100:00080000:0.0:1769077613.032829:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077613.032834:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077613.032837:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077613.032844:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077613.032858:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616766208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077613.032867:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077613.032870:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077613.032873:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077613.032877:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4598 00000100:00080000:0.0:1769077613.032881:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077613.032882:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077613/real 1769077613] req@ffff995b84641880 x1855007616765824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077613.032890:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077613.032900:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077613.032915:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616765824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077613.032917:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616766336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077613.032951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077613.032951:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077613.032960:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077613.032963:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077613.032964:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077613.032972:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616765824:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077613.032973:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4598 00000100:00080000:0.0:1769077613.032980:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077613.033004:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077613.033010:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077613.033022:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616766464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077613.033028:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765952:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077613.033041:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077613.033045:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077613.033047:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077613.033053:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4598 00000100:00080000:0.0:1769077613.033057:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077613.033062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077613.033064:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077613.033073:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077613.033084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077613/real 1769077613] req@ffff995b84640700 x1855007616765952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077613.033089:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616766592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077613.033125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616765952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077613.033134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077613.033138:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077613.033142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616765952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077613.033148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077613.033166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077613.033177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077613/real 1769077613] req@ffff995b84642a00 x1855007616766080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077613.033189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616766080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077613.033199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077613.033202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077613.033206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077613.033221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766208:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077613.033239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077613.033250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077613/real 1769077613] req@ffff995b84643b80 x1855007616766208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077613.033261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616766208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077613.033276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077613.033279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077613.033282:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766208:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077613.033288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766336:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077613.033303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077613.033313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077613/real 1769077613] req@ffff995b84643100 x1855007616766336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077613.033323:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616766336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077613.033332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077613.033335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077613.033339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077613.033357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616766464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077613.033373:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077613.033384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077613/real 1769077613] req@ffff995b84642d80 x1855007616766464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077613.033395:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616766464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077613.033403:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077613.033407:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077613.033410:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616766464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077613.033416:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616766592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077613.033431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077613.033440:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077613/real 1769077613] req@ffff995b84643480 x1855007616766592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077613.033474:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616766592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077613.033485:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077613.033491:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077613.033495:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616766592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077613.736533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077613.736581:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077614.056570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077614.056576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077614.056594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077614.056617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077614.760550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077614.760630:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077615.080545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077615.080562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077615.080565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077615.080584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077615.784517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077615.784601:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077616.104562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077616.104567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077616.104568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077616.104705:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077616.808539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077616.808598:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077617.128556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077617.128565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077617.128581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077617.128609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077617.832574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077617.832636:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077618.152553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077618.152553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077618.152573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077618.152579:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077618.152587:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077618.152592:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077618.152602:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4603 00000100:00100000:1.0:1769077618.152606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077618.152614:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077618.152622:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077618.152669:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616766720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077618.152703:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077618.152707:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077618.152709:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077618.152714:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4603 00000100:00080000:0.0:1769077618.152719:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077618.152723:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077618.152732:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077618.152743:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616766848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077618.152751:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077618.152754:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077618.152756:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077618.152760:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4603 00000100:00080000:0.0:1769077618.152764:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077618.152769:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077618.152774:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077618.152779:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616766720:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077618.152790:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616766976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077618.152803:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077618.152808:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077618.152811:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077618.152815:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4603 00000100:00080000:0.0:1769077618.152820:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077618.152823:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077618.152828:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077618.152839:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616767104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077618.152843:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077618.152855:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077618.152859:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077618.152861:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077618.152867:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4603 00000100:00080000:0.0:1769077618.152871:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077618.152876:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077618.152881:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077618.152891:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077618/real 1769077618] req@ffff995b84643800 x1855007616766720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077618.152909:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616767232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077618.152919:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077618.152924:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077618.152927:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077618.152934:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4603 00000100:00080000:0.0:1769077618.152939:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077618.152943:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077618.152948:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077618.152950:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616766720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077618.152959:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616767360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077618.152965:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077618.152968:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077618.152972:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077618.152975:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077618.152976:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077618.152985:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4603 00000100:00100000:3.0:1769077618.152986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616766720:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077618.152988:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077618.152993:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077618.152997:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077618.152997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077618.153008:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616767488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077618.153024:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077618.153037:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077618/real 1769077618] req@ffff995b84641180 x1855007616766848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077618.153045:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616766848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077618.153051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077618.153053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077618.153056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077618.153076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077618.153086:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077618.153093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077618/real 1769077618] req@ffff995b84641500 x1855007616766976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077618.153100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616766976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077618.153108:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077618.153109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077618.153111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616766976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077618.153115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616767104:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077618.153125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077618.153131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077618/real 1769077618] req@ffff995b84642680 x1855007616767104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077618.153138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616767104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077618.153144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077618.153146:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077618.153148:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616767104:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077618.153151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616767232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077618.153160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077618.153166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077618/real 1769077618] req@ffff995b84640e00 x1855007616767232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077618.153173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616767232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077618.153179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077618.153181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077618.153182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616767232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077618.153186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616767360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077618.153195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077618.153200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077618/real 1769077618] req@ffff995b84641c00 x1855007616767360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077618.153212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616767360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077618.153217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077618.153219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077618.153221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616767360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077618.153224:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616767488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077618.153233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077618.153239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077618/real 1769077618] req@ffff995b84640a80 x1855007616767488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077618.153247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616767488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077618.153252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077618.153254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077618.153256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616767488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077618.856555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077618.856608:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077619.176561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077619.176564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077619.176591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077619.176616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077619.880573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077619.880646:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077620.200564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077620.200564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077620.200564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077620.200570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077620.904560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077620.904628:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077621.224562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077621.224562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077621.224562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077621.224563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077621.928573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077621.928639:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077622.248575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077622.248575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077622.248577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077622.248638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077622.952543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077622.952606:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077623.272579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077623.272579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077623.272611:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077623.272618:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077623.272623:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077623.272632:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4608 00000100:00080000:0.0:1769077623.272644:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077623.272651:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077623.272677:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077623.272693:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616767616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077623.272720:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077623.272724:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077623.272727:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077623.272732:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4608 00000100:00100000:1.0:1769077623.272736:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616767616:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077623.272738:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077623.272747:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077623.272758:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077623.272775:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616767744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077623.272786:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077623.272789:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077623.272791:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077623.272794:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4608 00000100:00080000:0.0:1769077623.272797:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077623.272801:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077623.272805:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077623.272831:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077623.272844:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616767872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077623.272851:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077623.272854:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077623.272856:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077623.272860:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4608 00000100:00080000:0.0:1769077623.272864:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077623.272867:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077623/real 1769077623] req@ffff995b84640380 x1855007616767616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077623.272870:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077623.272871:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000008:00100000:0.0:1769077623.272879:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077623.272893:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616768000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077623.272901:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616767616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077623.272902:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077623.272909:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077623.272912:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077623.272920:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4608 00000100:00080000:0.0:1769077623.272923:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077623.272925:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077623.272928:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769077623.272932:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077623.272937:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616768128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077623.272941:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077623.272950:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616767616:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077623.272952:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077623.272957:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077623.272958:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077623.272962:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4608 00000100:00080000:0.0:1769077623.272964:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077623.272967:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077623.272969:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077623.272976:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616768256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077623.272978:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616767744:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077623.272991:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077623.272994:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077623.272996:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077623.272999:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4608 00000100:00080000:0.0:1769077623.273004:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077623.273008:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769077623.273009:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077623.273012:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077623.273022:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616768384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077623.273025:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077623/real 1769077623] req@ffff995b84641f80 x1855007616767744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077623.273051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616767744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077623.273061:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077623.273065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077623.273070:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616767744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077623.273077:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616767872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077623.273099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077623.273119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077623/real 1769077623] req@ffff995b84640000 x1855007616767872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077623.273130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616767872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077623.273138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077623.273141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077623.273143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616767872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077623.273147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768000:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077623.273163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077623.273171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077623/real 1769077623] req@ffff995b84640a80 x1855007616768000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077623.273178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616768000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077623.273187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077623.273190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077623.273194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077623.273200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768128:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077623.273219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077623.273231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077623/real 1769077623] req@ffff995b84641c00 x1855007616768128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077623.273244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616768128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077623.273252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077623.273254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077623.273257:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077623.273271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616768256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077623.273286:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077623.273293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077623/real 1769077623] req@ffff995b84640e00 x1855007616768256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077623.273306:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616768256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077623.273312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077623.273314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077623.273316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616768256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077623.273320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616768384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077623.273329:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077623.273335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077623/real 1769077623] req@ffff995b84642680 x1855007616768384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077623.273342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616768384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077623.273347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077623.273348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077623.273350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616768384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077623.976561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077623.976632:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077624.296546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077624.296554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077624.296580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077624.296584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077625.000561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077625.000632:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077625.320578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077625.320578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077625.320585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077625.320713:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077626.024546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077626.024602:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077626.344533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077626.344542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077626.344544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077626.344572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077627.048513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077627.048620:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077627.368560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077627.368565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077627.368568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077627.368591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077628.072557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077628.072621:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077628.392520:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077628.392533:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077628.392545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077628.392550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077628.392570:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077628.392575:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077628.392580:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077628.392588:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4614 00000100:00080000:3.0:1769077628.392598:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077628.392604:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077628.392641:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616768512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077628.392663:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077628.392666:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077628.392667:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077628.392671:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4614 00000100:00080000:3.0:1769077628.392674:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077628.392677:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077628.392686:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077628.392693:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616768512:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077628.392697:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616768640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077628.392708:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077628.392712:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077628.392715:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077628.392719:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4614 00000100:00080000:3.0:1769077628.392735:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077628.392741:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077628.392747:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077628.392761:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616768768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077628.392772:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077628.392778:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1769077628.392779:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077628.392781:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077628.392789:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4614 00000100:00080000:3.0:1769077628.392794:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077628.392798:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077628.392803:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077628.392814:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616768896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077628.392821:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077628.392824:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077628.392828:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077628.392829:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077628/real 1769077628] req@ffff995b84558000 x1855007616768512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077628.392832:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4614 00000100:00080000:3.0:1769077628.392837:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077628.392849:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077628.392854:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077628.392864:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616768512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077628.392864:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616769024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077628.392897:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077628.392898:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077628.392904:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077628.392907:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077628.392908:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077628.392913:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4614 00000100:00100000:1.0:1769077628.392915:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616768512:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077628.392917:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077628.392924:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077628.392927:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077628.392934:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616769152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077628.392942:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077628.392944:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768640:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077628.392946:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077628.392951:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077628.392956:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4614 00000100:00080000:3.0:1769077628.392959:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077628.392961:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077628.392965:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077628.392976:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077628.392977:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616769280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077628.392994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077628/real 1769077628] req@ffff995b84559880 x1855007616768640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077628.393006:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616768640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077628.393013:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077628.393016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077628.393018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077628.393023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077628.393035:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077628.393041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077628/real 1769077628] req@ffff995b84559c00 x1855007616768768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077628.393054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616768768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077628.393061:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077628.393063:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077628.393065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077628.393068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077628.393079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077628.393085:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077628/real 1769077628] req@ffff995b8455a300 x1855007616768896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077628.393092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616768896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077628.393098:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077628.393100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077628.393102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616768896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077628.393105:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769024:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077628.393114:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077628.393120:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077628/real 1769077628] req@ffff995b8455ad80 x1855007616769024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077628.393128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616769024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077628.393148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077628.393150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077628.393152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077628.393163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616769152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077628.393183:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077628.393193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077628/real 1769077628] req@ffff995b8455b480 x1855007616769152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077628.393201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616769152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077628.393207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077628.393209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077628.393211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616769152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077628.393215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616769280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077628.393226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077628.393232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077628/real 1769077628] req@ffff995b84558700 x1855007616769280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077628.393240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616769280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077628.393245:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077628.393247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077628.393250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616769280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077629.096528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077629.096606:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077629.416558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077629.416566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077629.416567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077629.416591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077630.120548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077630.120616:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077630.440564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077630.440570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077630.440574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077630.440598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077631.144533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077631.144611:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077631.464567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077631.464573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077631.464579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077631.464638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077632.168524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077632.168571:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077632.488532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077632.488536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077632.488559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077632.488596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077633.192536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077633.192581:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077633.512568:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077633.512574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077633.512578:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769077633.512582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077633.512584:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077633.512587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077633.512596:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4619 00000100:00080000:1.0:1769077633.512607:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077633.512614:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077633.512620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077633.512676:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616769408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077633.512699:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077633.512702:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077633.512703:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077633.512707:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4619 00000100:00080000:1.0:1769077633.512709:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077633.512713:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077633.512721:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077633.512732:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616769536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077633.512737:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616769408:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077633.512745:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077633.512749:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077633.512750:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077633.512753:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4619 00000100:00080000:1.0:1769077633.512755:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077633.512758:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077633.512761:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077633.512769:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616769664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077633.512779:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077633.512783:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077633.512785:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077633.512788:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4619 00000100:00080000:1.0:1769077633.512792:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077633.512796:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077633.512801:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077633.512813:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616769792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077633.512822:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077633.512824:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077633.512829:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077633.512831:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077633.512835:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4619 00000100:00080000:1.0:1769077633.512840:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077633.512845:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077633.512851:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077633.512865:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077633/real 1769077633] req@ffff995b868c6300 x1855007616769408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077633.512867:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616769920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077633.512898:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077633.512902:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077633.512904:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077633.512910:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616769408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077633.512911:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4619 00000100:00080000:1.0:1769077633.512916:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077633.512924:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077633.512932:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077633.512933:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077633.512943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077633.512951:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616770048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077633.512958:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616769408:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077633.512964:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077633.512970:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077633.512972:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077633.512979:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4619 00000100:00080000:1.0:1769077633.512983:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077633.512986:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077633.512991:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769536:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769077633.512993:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077633.513007:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616770176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077633.513029:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077633.513042:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077633/real 1769077633] req@ffff995b868c6d80 x1855007616769536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077633.513052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616769536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077633.513058:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077633.513061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077633.513063:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077633.513067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077633.513080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077633.513086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077633/real 1769077633] req@ffff995b868c4a80 x1855007616769664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077633.513094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616769664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077633.513102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077633.513104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077633.513106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077633.513110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077633.513120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077633.513126:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077633/real 1769077633] req@ffff995b868c4700 x1855007616769792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077633.513133:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616769792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077633.513139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077633.513141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077633.513143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077633.513146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077633.513155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077633.513160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077633/real 1769077633] req@ffff995b868c5500 x1855007616769920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077633.513168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616769920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077633.513178:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077633.513180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077633.513182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616769920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077633.513185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616770048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077633.513195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077633.513200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077633/real 1769077633] req@ffff995b868c7800 x1855007616770048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077633.513207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616770048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077633.513212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077633.513213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077633.513215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616770048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077633.513229:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616770176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077633.513248:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077633.513259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077633/real 1769077633] req@ffff995b868c7480 x1855007616770176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077633.513271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616770176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077633.513280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077633.513283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077633.513287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616770176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077634.216575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077634.216665:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077634.536595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077634.536595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077634.536595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077634.536695:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077635.240532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077635.240606:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077635.560590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077635.560590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077635.560593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077635.560663:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077636.264554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077636.264616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077636.584566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077636.584570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077636.584576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077636.584579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077637.288565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077637.288653:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077637.608561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077637.608561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077637.608563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077637.608563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077638.312534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077638.312596:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077638.632638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077638.632639:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077638.632647:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077638.632657:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077638.632664:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077638.632679:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4624 00000100:00100000:3.0:1769077638.632701:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077638.632702:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077638.632719:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077638.632728:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077638.632767:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616770304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077638.632807:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077638.632811:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077638.632812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077638.632816:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4624 00000100:00080000:2.0:1769077638.632822:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077638.632826:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077638.632839:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077638.632860:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616770304:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077638.632861:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616770432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077638.632886:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077638.632890:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077638.632893:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077638.632902:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4624 00000100:00080000:2.0:1769077638.632909:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077638.632917:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077638.632924:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077638.632933:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077638.632947:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616770560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077638.632960:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077638.632962:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077638.632963:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077638.632967:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4624 00000100:00080000:2.0:1769077638.632970:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077638.632976:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077638.632977:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077638/real 1769077638] req@ffff995b85acaa00 x1855007616770304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077638.632980:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077638.633001:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616770688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077638.633003:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616770304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077638.633023:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077638.633026:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077638.633027:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077638.633031:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077638.633035:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077638.633038:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4624 00000100:00100000:1.0:1769077638.633042:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616770304:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077638.633042:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077638.633047:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077638.633052:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077638.633061:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616770816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077638.633070:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077638.633072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616770432:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077638.633073:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077638.633077:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077638.633082:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4624 00000100:00080000:2.0:1769077638.633085:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077638.633089:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077638.633092:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077638.633098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077638.633104:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616770944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077638.633112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077638/real 1769077638] req@ffff995b85aca300 x1855007616770432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077638.633114:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077638.633120:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077638.633123:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077638.633131:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4624 00000100:00100000:1.0:1769077638.633136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616770432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077638.633139:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077638.633147:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769077638.633149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769077638.633151:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769077638.633155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077638.633160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616770432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077638.633164:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616771072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077638.633174:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616770560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077638.633193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077638.633203:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077638/real 1769077638] req@ffff995b85ac8380 x1855007616770560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077638.633211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616770560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077638.633219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077638.633221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077638.633224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616770560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077638.633229:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616770688:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077638.633242:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077638.633249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077638/real 1769077638] req@ffff995b85acb100 x1855007616770688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077638.633258:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616770688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077638.633265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077638.633268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077638.633270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616770688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077638.633281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616770816:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077638.633294:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077638.633304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077638/real 1769077638] req@ffff995b85ac9c00 x1855007616770816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077638.633312:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616770816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077638.633320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077638.633323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077638.633325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616770816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077638.633329:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616770944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077638.633343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077638.633367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077638/real 1769077638] req@ffff995b85ac9880 x1855007616770944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077638.633385:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616770944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077638.633395:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077638.633399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077638.633402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616770944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077638.633408:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616771072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077638.633429:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077638.633440:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077638/real 1769077638] req@ffff995b85acbb80 x1855007616771072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077638.633452:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616771072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077638.633499:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077638.633504:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077638.633511:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616771072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077639.336561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077639.336622:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077639.656596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077639.656596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077639.656626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077639.656632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077640.360572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077640.360630:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077640.680569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077640.680569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077640.680570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077640.680583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077641.384584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077641.384641:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077641.704583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077641.704583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077641.704588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077641.704606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077642.408586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077642.408653:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077642.728615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077642.728616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077642.728650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077642.728651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077643.432577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077643.432643:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077643.752540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077643.752566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077643.752571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077643.752572:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077643.752579:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769077643.752580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077643.752584:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077643.752593:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4629 00000100:00080000:3.0:1769077643.752604:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077643.752610:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077643.752655:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616771200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077643.752676:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077643.752679:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077643.752680:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077643.752683:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4629 00000100:00080000:3.0:1769077643.752686:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077643.752689:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077643.752699:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077643.752708:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616771200:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077643.752716:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616771328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077643.752725:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077643.752728:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077643.752729:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077643.752732:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4629 00000100:00080000:3.0:1769077643.752735:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077643.752737:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077643.752740:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077643.752748:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616771456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077643.752754:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077643.752757:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077643.752758:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077643.752767:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4629 00000100:00080000:3.0:1769077643.752770:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077643.752774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077643.752775:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077643.752780:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077643.752789:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616771584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077643.752795:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077643.752797:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077643.752799:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077643.752802:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4629 00000100:00080000:3.0:1769077643.752805:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077643.752808:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077643.752812:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077643.752813:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077643/real 1769077643] req@ffff995b84558700 x1855007616771200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077643.752823:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616771712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077643.752830:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077643.752833:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616771200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077643.752834:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077643.752839:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077643.752848:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4629 00000100:00080000:1.0:1769077643.752851:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077643.752852:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077643.752859:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769077643.752861:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769077643.752864:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077643.752867:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616771200:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077643.752875:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616771840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077643.752881:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077643.752884:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077643.752886:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077643.752891:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616771328:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077643.752891:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4629 00000100:00080000:3.0:1769077643.752893:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077643.752897:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077643.752901:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077643.752912:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616771968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077643.752919:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077643.752936:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077643/real 1769077643] req@ffff995b8455b480 x1855007616771328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077643.752946:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616771328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077643.752952:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077643.752954:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077643.752957:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616771328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077643.752961:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616771456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077643.752971:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077643.752977:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077643/real 1769077643] req@ffff995b8455ad80 x1855007616771456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077643.752985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616771456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077643.752991:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077643.752993:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077643.752995:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616771456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077643.752998:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616771584:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077643.753009:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077643.753014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077643/real 1769077643] req@ffff995b8455a300 x1855007616771584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077643.753022:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616771584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077643.753036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077643.753041:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077643.753043:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616771584:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077643.753047:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616771712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077643.753059:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077643.753066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077643/real 1769077643] req@ffff995b84559c00 x1855007616771712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077643.753073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616771712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077643.753079:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077643.753081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077643.753083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616771712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077643.753086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616771840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077643.753096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077643.753102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077643/real 1769077643] req@ffff995b84559880 x1855007616771840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077643.753109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616771840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077643.753115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077643.753117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077643.753118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616771840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077643.753129:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616771968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077643.753139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077643.753145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077643/real 1769077643] req@ffff995b84558000 x1855007616771968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077643.753157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616771968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077643.753162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077643.753164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077643.753165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616771968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077644.456542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077644.456620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077644.776626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077644.776641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077644.776642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077644.776672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077645.480556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077645.480610:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077645.800548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077645.800559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077645.800578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077645.800691:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077646.504551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077646.504614:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077646.824609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077646.824610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077646.824629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077646.824652:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077647.528580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077647.528679:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077647.848527:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077647.848545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077647.848589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077647.848592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077648.552534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077648.552584:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077648.872561:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077648.872570:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077648.872575:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077648.872580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077648.872584:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4634 00000100:00080000:3.0:1769077648.872597:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077648.872599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077648.872605:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077648.872617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077648.872631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077648.872645:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616772096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077648.872670:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077648.872673:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077648.872674:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077648.872684:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4634 00000100:00080000:3.0:1769077648.872687:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077648.872691:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077648.872700:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077648.872709:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616772224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077648.872718:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616772096:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077648.872719:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077648.872725:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077648.872729:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077648.872733:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4634 00000100:00080000:3.0:1769077648.872735:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077648.872738:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077648.872741:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077648.872753:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616772352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077648.872763:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077648.872767:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077648.872769:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077648.872775:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4634 00000100:00080000:3.0:1769077648.872779:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077648.872783:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077648.872788:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769077648.872799:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077648.872803:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616772480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077648.872816:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077648.872820:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077648.872822:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077648.872826:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4634 00000100:00080000:3.0:1769077648.872830:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077648.872834:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077648.872840:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077648.872850:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077648/real 1769077648] req@ffff995b8455aa00 x1855007616772096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077648.872862:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616772608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077648.872874:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077648.872876:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616772096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077648.872883:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077648.872888:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077648.872898:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077648.872898:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4634 00000100:00080000:3.0:1769077648.872908:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077648.872910:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077648.872916:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077648.872920:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616772096:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769077648.872925:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077648.872943:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616772736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077648.872955:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077648.872957:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616772224:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077648.872959:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077648.872963:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077648.872972:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4634 00000100:00080000:3.0:1769077648.872976:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077648.872978:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077648.872983:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077648.872993:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616772864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077648.873002:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077648.873023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077648/real 1769077648] req@ffff995b8455b100 x1855007616772224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077648.873039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616772224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077648.873049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077648.873053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077648.873057:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616772224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077648.873064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616772352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077648.873085:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077648.873097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077648/real 1769077648] req@ffff995b8455a680 x1855007616772352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077648.873110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616772352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077648.873121:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077648.873124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077648.873128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616772352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077648.873135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616772480:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077648.873156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077648.873168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077648/real 1769077648] req@ffff995b84558a80 x1855007616772480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077648.873181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616772480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077648.873190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077648.873193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077648.873197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616772480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077648.873203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616772608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077648.873235:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077648.873248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077648/real 1769077648] req@ffff995b8455bb80 x1855007616772608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077648.873262:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616772608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077648.873271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077648.873275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077648.873278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616772608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077648.873296:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616772736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077648.873317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077648.873329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077648/real 1769077648] req@ffff995b84559500 x1855007616772736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077648.873342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616772736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077648.873357:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077648.873360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077648.873363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616772736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077648.873369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616772864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077648.873424:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077648.873439:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077648/real 1769077648] req@ffff995b8455aa00 x1855007616772864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077648.873476:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616772864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077648.873487:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077648.873491:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077648.873494:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616772864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077649.576615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077649.576677:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077649.896569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077649.896569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077649.896579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077649.896610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077650.600539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077650.600618:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077650.920581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077650.920581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077650.920581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077650.920596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077651.624563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077651.624627:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077651.944614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077651.944618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077651.944620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077651.944677:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077652.648528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077652.648599:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077652.968567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077652.968567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077652.968602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077652.968633:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077653.672554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077653.672606:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077653.992572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077653.992572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077653.992584:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077653.992593:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077653.992598:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077653.992604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077653.992608:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4639 00000100:00080000:1.0:1769077653.992621:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077653.992627:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077653.992628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077653.992668:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616772992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077653.992692:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077653.992695:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077653.992696:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077653.992699:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4639 00000100:00080000:1.0:1769077653.992702:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077653.992706:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077653.992714:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077653.992720:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616772992:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077653.992726:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616773120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077653.992734:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077653.992736:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077653.992738:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077653.992741:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4639 00000100:00080000:1.0:1769077653.992743:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077653.992745:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077653.992748:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077653.992757:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616773248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077653.992762:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077653.992765:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077653.992766:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077653.992769:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4639 00000100:00080000:1.0:1769077653.992773:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077653.992776:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077653.992778:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077653.992784:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077653.992789:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616773376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077653.992796:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077653.992799:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077653.992800:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077653.992804:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4639 00000100:00080000:1.0:1769077653.992806:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077653.992808:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077653.992811:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077653.992819:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616773504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077653.992823:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077653/real 1769077653] req@ffff995b868c5c00 x1855007616772992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077653.992830:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077653.992837:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077653.992840:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077653.992846:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4639 00000100:00080000:1.0:1769077653.992848:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077653.992852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616772992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077653.992855:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077653.992861:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077653.992870:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077653.992874:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616773632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077653.992878:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077653.992886:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616772992:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077653.992889:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077653.992893:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077653.992894:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077653.992898:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4639 00000100:00080000:1.0:1769077653.992901:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077653.992903:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077653.992906:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077653.992912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616773120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077653.992916:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616773760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077653.992935:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077653.992948:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077653/real 1769077653] req@ffff995b868c6a00 x1855007616773120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077653.992970:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616773120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077653.992976:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077653.992979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077653.992981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616773120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077653.992985:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616773248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077653.992997:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077653.993004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077653/real 1769077653] req@ffff995b868c7100 x1855007616773248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077653.993012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616773248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077653.993019:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077653.993021:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077653.993023:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616773248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077653.993027:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616773376:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077653.993037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077653.993043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077653/real 1769077653] req@ffff995b868c7480 x1855007616773376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077653.993051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616773376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077653.993057:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077653.993059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077653.993061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616773376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077653.993064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616773504:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077653.993073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077653.993078:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077653/real 1769077653] req@ffff995b868c7800 x1855007616773504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077653.993086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616773504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077653.993096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077653.993098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077653.993100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616773504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077653.993104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616773632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077653.993112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077653.993118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077653/real 1769077653] req@ffff995b868c5500 x1855007616773632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077653.993125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616773632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077653.993130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077653.993132:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077653.993134:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616773632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077653.993144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616773760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077653.993154:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077653.993159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077653/real 1769077653] req@ffff995b868c4700 x1855007616773760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077653.993166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616773760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077653.993171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077653.993173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077653.993175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616773760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077654.696541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077654.696593:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077655.016580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077655.016580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077655.016580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077655.016751:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077655.720540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077655.720604:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077656.040554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077656.040568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077656.040583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077656.040595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077656.744571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077656.744639:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077657.064566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077657.064569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077657.064569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077657.064572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077657.768563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077657.768626:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077658.088568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077658.088569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077658.088574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077658.088585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077658.792559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077658.792634:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077659.112590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077659.112591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077659.112594:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077659.112607:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077659.112612:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077659.112615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077659.112624:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4644 00000100:00080000:1.0:1769077659.112639:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077659.112648:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077659.112662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077659.112690:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616773888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077659.112716:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077659.112719:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077659.112720:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077659.112723:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4644 00000100:00080000:1.0:1769077659.112726:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077659.112729:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077659.112738:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077659.112747:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616774016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077659.112758:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077659.112763:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616773888:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077659.112764:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077659.112771:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077659.112826:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4644 00000100:00080000:1.0:1769077659.112830:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077659.112833:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077659.112837:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077659.112850:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616774144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077659.112861:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077659.112872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077659.112876:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077659.112878:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077659.112884:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4644 00000100:00080000:1.0:1769077659.112891:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077659.112894:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077659.112899:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077659.112909:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616774272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077659.112917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077659.112920:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077659/real 1769077659] req@ffff995b868c4a80 x1855007616773888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077659.112921:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077659.112927:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077659.112942:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4644 00000100:00080000:1.0:1769077659.112954:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077659.112960:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077659.112964:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077659.112970:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616773888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077659.112981:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616774400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077659.112990:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077659.113005:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077659.113020:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616773888:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077659.113024:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077659.113029:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077659.113031:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077659.113035:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4644 00000100:00080000:1.0:1769077659.113038:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077659.113041:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077659.113045:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077659.113053:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616774528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077659.113061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774016:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077659.113085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077659.113092:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077659.113094:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077659.113099:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4644 00000100:00080000:1.0:1769077659.113103:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077659.113111:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769077659.113115:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769077659.113119:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077659.113133:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616774656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077659.113137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077659/real 1769077659] req@ffff995b868c6d80 x1855007616774016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077659.113198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616774016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077659.113220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077659.113227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077659.113233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077659.113240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077659.113259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077659.113268:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077659/real 1769077659] req@ffff995b868c6300 x1855007616774144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077659.113277:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616774144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077659.113284:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077659.113287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077659.113290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077659.113294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077659.113306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077659.113312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077659/real 1769077659] req@ffff995b868c4700 x1855007616774272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077659.113319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616774272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077659.113327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077659.113329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077659.113330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077659.113334:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077659.113349:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077659.113360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077659/real 1769077659] req@ffff995b868c5500 x1855007616774400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077659.113373:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616774400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077659.113384:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077659.113387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077659.113391:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077659.113412:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616774528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077659.113432:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077659.113441:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077659/real 1769077659] req@ffff995b868c7800 x1855007616774528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077659.113449:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616774528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077659.113476:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077659.113479:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077659.113481:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616774528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077659.113486:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616774656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077659.113499:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077659.113507:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077659/real 1769077659] req@ffff995b868c7480 x1855007616774656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077659.113522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616774656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077659.113531:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077659.113534:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077659.113538:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616774656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077659.816559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077659.816716:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077660.136560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077660.136565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077660.136613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077660.136631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077660.840577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077660.840641:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077661.160557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077661.160557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077661.160568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077661.160575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077661.864539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077661.864621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077662.184533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077662.184554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077662.184556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077662.184609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077662.888558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077662.888638:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077663.208553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077663.208553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077663.208572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077663.208590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077663.912528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077663.912611:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077664.232582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077664.232606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077664.232630:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077664.232651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077664.232658:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077664.232667:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077664.232673:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077664.232686:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4649 00000100:00080000:1.0:1769077664.232699:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077664.232748:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077664.232786:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616774784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077664.232808:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077664.232812:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077664.232814:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077664.232817:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4649 00000100:00080000:1.0:1769077664.232820:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077664.232825:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077664.232833:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077664.232842:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616774912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077664.232851:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077664.232853:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077664.232854:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077664.232860:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4649 00000100:00100000:3.0:1769077664.232862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616774784:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077664.232863:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077664.232873:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077664.232884:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077664.232890:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616775040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077664.232896:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077664.232898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077664.232900:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077664.232904:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4649 00000100:00080000:1.0:1769077664.232907:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077664.232909:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077664.232914:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077664.232925:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616775168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077664.232932:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077664.232935:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077664.232936:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077664.232938:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4649 00000100:00080000:1.0:1769077664.232943:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077664.232945:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077664.232948:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077664.232954:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616775296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077664.232961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077664.232964:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077664.232965:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077664.232968:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4649 00000100:00080000:1.0:1769077664.232970:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077664.232973:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077664.232975:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077664.232982:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616775424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077664.232989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077664.232993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077664.232995:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077664.232999:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4649 00000100:00080000:1.0:1769077664.233002:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077664.233005:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077664.233010:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077664.233023:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616775552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077664.233040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077664.233079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077664.233098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077664.233115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077664.233166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616775424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077664.233199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616775552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077664.936601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077664.936675:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077665.256534:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077665.256536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077665.256560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077665.256580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077665.960560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077665.960620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077666.280569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077666.280569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077666.280637:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077666.280951:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077666.984563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077666.984639:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077667.304563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077667.304563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077667.304571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077667.304608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769077667.304791:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769077667.304837:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077667.304858:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077667.304898:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077667.304914:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769077667.304922:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077664/real 1769077667] req@ffff995b868c7100 x1855007616774784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1769077667.304937:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769077667.304952:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769077667.304958:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616774784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1769077667.304967:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00080000:2.0:1769077667.304979:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:1.0:1769077667.304986:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00080000:2.0:1769077667.304991:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077667.304999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616774784:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077667.305023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077664/real 1769077667] req@ffff995b868c6a00 x1855007616774912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077667.305039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616774912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077667.305047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077667.305050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077667.305054:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616774912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077667.305065:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077664/real 1769077667] req@ffff995b868c5c00 x1855007616775040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077667.305111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616775040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077667.305123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077667.305127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077667.305131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077667.305139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077664/real 1769077667] req@ffff995b868c7480 x1855007616775168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077667.305151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616775168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077667.305161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077667.305165:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077667.305169:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077667.305176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077664/real 1769077667] req@ffff995b868c7800 x1855007616775296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077667.305188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616775296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077667.305197:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077667.305200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077667.305204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077667.305211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077664/real 1769077667] req@ffff995b868c5500 x1855007616775424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077667.305224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616775424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077667.305232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077667.305236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077667.305240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616775424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077667.305247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077664/real 1769077667] req@ffff995b868c4700 x1855007616775552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077667.305259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616775552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077667.305266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077667.305270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077667.305273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616775552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077668.008561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077668.008614:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077668.328575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077668.328616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077668.328616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077668.328668:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077669.032554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077669.032637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077669.352544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077669.352549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077669.352588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077669.352590:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077669.352597:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077669.352602:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077669.352609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077669.352614:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4655 00000100:00080000:3.0:1769077669.352627:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077669.352635:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077669.352676:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616775680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077669.352699:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077669.352702:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077669.352703:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077669.352707:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4655 00000100:00080000:3.0:1769077669.352711:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077669.352715:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077669.352724:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077669.352728:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616775680:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077669.352739:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616775808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077669.352751:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077669.352755:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077669.352757:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077669.352761:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4655 00000100:00080000:3.0:1769077669.352778:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077669.352784:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077669.352790:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077669.352792:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077669.352810:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616775936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077669.352822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077669.352825:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077669.352827:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077669.352833:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4655 00000100:00100000:2.0:1769077669.352834:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077669/real 1769077669] req@ffff995b84558000 x1855007616775680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077669.352837:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077669.352845:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077669.352855:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077669.352863:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616775680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077669.352872:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616776064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077669.352879:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077669.352888:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077669.352892:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077669.352897:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616775680:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077669.352903:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077669.352907:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077669.352912:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4655 00000100:00080000:3.0:1769077669.352917:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077669.352920:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077669.352925:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077669.352937:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616776192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077669.352940:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775808:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077669.352967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077669.352970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077669.352972:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077669.352978:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4655 00000800:00000100:2.0:1769077669.352979:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077669.352984:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077669.352991:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077669.352998:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077669.352999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077669/real 1769077669] req@ffff995b84559880 x1855007616775808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077669.353018:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616776320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077669.353020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616775808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077669.353062:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077669.353066:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077669.353072:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077669.353076:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077669.353081:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4655 00000100:00080000:2.0:1769077669.353087:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077669.353089:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077669.353095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775808:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077669.353100:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077669.353109:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077669.353110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077669.353126:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616776448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077669.353143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077669.353159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077669/real 1769077669] req@ffff995b84559c00 x1855007616775936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077669.353172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616775936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077669.353184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077669.353188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077669.353192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616775936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077669.353200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776064:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077669.353223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077669.353237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077669/real 1769077669] req@ffff995b8455a300 x1855007616776064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077669.353250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616776064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077669.353277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077669.353281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077669.353284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776064:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077669.353302:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077669.353321:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077669.353333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077669/real 1769077669] req@ffff995b8455ad80 x1855007616776192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077669.353344:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616776192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077669.353353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077669.353356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077669.353359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077669.353365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616776320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077669.353390:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077669.353402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077669/real 1769077669] req@ffff995b8455b480 x1855007616776320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077669.353414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616776320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077669.353422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077669.353425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077669.353428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616776320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077669.353434:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616776448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077669.353452:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077669.353490:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077669/real 1769077669] req@ffff995b84558700 x1855007616776448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077669.353503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616776448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077669.353509:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077669.353512:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077669.353514:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616776448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077670.056551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077670.056600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077670.376572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077670.376572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077670.376644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077670.376707:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077671.080609:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077671.080749:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077671.400545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077671.400552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077671.400570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077671.400592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077672.104530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077672.104600:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077672.424562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077672.424562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077672.424568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077672.424580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077673.128554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077673.128615:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077673.448566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077673.448566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077673.448567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077673.448572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077674.152584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077674.152646:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077674.472565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077674.472565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077674.472571:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077674.472577:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769077674.472582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077674.472583:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077674.472594:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4660 00000100:00080000:1.0:1769077674.472605:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077674.472611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077674.472612:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077674.472661:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616776576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077674.472681:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077674.472684:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077674.472685:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077674.472688:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4660 00000100:00080000:1.0:1769077674.472690:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077674.472694:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077674.472702:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077674.472714:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616776704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077674.472715:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616776576:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077674.472730:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077674.472733:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077674.472734:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077674.472737:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4660 00000100:00080000:1.0:1769077674.472743:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077674.472747:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077674.472753:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077674.472766:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616776832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077674.472777:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077674.472781:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077674.472783:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769077674.472787:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077674.472792:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4660 00000100:00080000:1.0:1769077674.472798:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077674.472803:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077674.472810:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077674.472824:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616776960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077674.472829:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077674/real 1769077674] req@ffff995b868c6300 x1855007616776576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077674.472843:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077674.472847:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616776576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077674.472848:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077674.472852:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077674.472859:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4660 00000100:00080000:1.0:1769077674.472865:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077674.472865:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077674.472877:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077674.472877:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769077674.472886:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077674.472887:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616776576:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077674.472897:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616777088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077674.472909:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776704:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077674.472912:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077674.472916:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077674.472918:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077674.472921:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4660 00000100:00080000:1.0:1769077674.472926:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077674.472929:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077674.472933:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077674.472940:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616777216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077674.472946:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077674.472968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077674.472971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077674.472973:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077674.472978:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4660 00000100:00100000:2.0:1769077674.472978:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077674/real 1769077674] req@ffff995b868c6d80 x1855007616776704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077674.472984:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077674.472992:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077674.472998:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077674.473001:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616776704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077674.473009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077674.473010:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616777344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077674.473017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077674.473026:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077674.473033:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077674.473053:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077674.473063:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077674/real 1769077674] req@ffff995b868c4a80 x1855007616776832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077674.473072:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616776832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077674.473080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077674.473082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077674.473084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077674.473088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776960:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077674.473102:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077674.473109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077674/real 1769077674] req@ffff995b868c4700 x1855007616776960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077674.473119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616776960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077674.473125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077674.473127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077674.473129:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616776960:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077674.473147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077674.473165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077674.473171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077674/real 1769077674] req@ffff995b868c5500 x1855007616777088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077674.473179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616777088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077674.473199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077674.473202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077674.473204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077674.473211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616777216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077674.473227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077674.473236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077674/real 1769077674] req@ffff995b868c7800 x1855007616777216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077674.473243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616777216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077674.473256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077674.473258:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077674.473260:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616777216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077674.473266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616777344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077674.473291:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077674.473304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077674/real 1769077674] req@ffff995b868c7480 x1855007616777344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077674.473320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616777344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077674.473330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077674.473335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077674.473342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616777344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077675.176540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077675.176598:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077675.496561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077675.496576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077675.496595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077675.496596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077676.200563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077676.200630:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077676.520525:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077676.520569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077676.520605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077676.520606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077677.224564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077677.224657:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077677.544519:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077677.544524:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077677.544549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077677.544552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077678.248563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077678.248654:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077678.568517:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077678.568547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077678.568570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077678.568588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077679.272543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077679.272626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077679.592532:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077679.592552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077679.592576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077679.592606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077679.592610:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077679.592616:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077679.592621:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077679.592632:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4665 00000100:00080000:2.0:1769077679.592643:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077679.592649:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077679.592689:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616777472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077679.592720:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077679.592723:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077679.592725:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077679.592728:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4665 00000100:00080000:2.0:1769077679.592731:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077679.592735:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077679.592744:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077679.592759:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616777600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077679.592761:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616777472:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077679.592782:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077679.592785:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077679.592786:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077679.592790:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4665 00000100:00080000:2.0:1769077679.592793:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077679.592795:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077679.592798:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077679.592816:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616777728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077679.592823:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077679.592826:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077679.592827:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077679.592830:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4665 00000100:00080000:2.0:1769077679.592833:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077679.592835:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077679.592838:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077679.592845:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616777856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077679.592851:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077679.592854:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077679.592855:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077679.592859:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4665 00000100:00080000:2.0:1769077679.592861:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077679.592876:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077679.592882:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077679.592890:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077679.592900:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616777984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077679.592911:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077679.592914:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077679.592916:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077679.592919:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4665 00000100:00080000:2.0:1769077679.592923:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077679.592926:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077679.592931:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077679.592950:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616778112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077679.592951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077679/real 1769077679] req@ffff995b85acb800 x1855007616777472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077679.592989:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077679.592992:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077679.592994:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077679.592997:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4665 00000100:00080000:2.0:1769077679.592999:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077679.593001:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077679.593005:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077679.593015:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616778240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077679.593106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616777472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077679.593116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077679.593119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077679.593122:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616777472:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077679.593145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077679.593191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077679.593221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077679/real 1769077679] req@ffff995b85ac8a80 x1855007616777600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077679.593230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616777600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077679.593236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077679.593239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077679.593241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077679.593246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077679.593258:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077679.593265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077679/real 1769077679] req@ffff995b85acbb80 x1855007616777728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077679.593272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616777728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077679.593279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077679.593281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077679.593283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077679.593286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777856:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077679.593297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077679.593302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077679/real 1769077679] req@ffff995b85ac9880 x1855007616777856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077679.593310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616777856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077679.593316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077679.593317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077679.593319:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077679.593323:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777984:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077679.593334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077679.593346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077679/real 1769077679] req@ffff995b85ac9c00 x1855007616777984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077679.593358:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616777984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077679.593368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077679.593372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077679.593375:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616777984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077679.593382:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616778112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077679.593398:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077679.593406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077679/real 1769077679] req@ffff995b85acb100 x1855007616778112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077679.593413:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616778112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077679.593419:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077679.593421:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077679.593423:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616778112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077679.593426:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616778240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077679.593436:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077679.593441:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077679/real 1769077679] req@ffff995b85ac8380 x1855007616778240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077679.593448:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616778240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077679.593483:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077679.593487:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077679.593491:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616778240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077680.296547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077680.296623:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077680.616536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077680.616539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077680.616542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077680.616550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077681.320557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077681.320623:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077681.640524:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077681.640524:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077681.640531:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077681.640582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077682.344564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077682.344624:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077682.664562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077682.664566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077682.664567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077682.664602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077683.368553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077683.368627:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077683.688549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077683.688549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077683.688552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077683.688570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077684.392552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077684.392636:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077684.712559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077684.712570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077684.712582:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077684.712602:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769077684.712602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077684.712608:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077684.712620:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4670 00000100:00100000:1.0:1769077684.712626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077684.712635:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077684.712646:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077684.712700:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616778368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077684.712744:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077684.712748:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077684.712751:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077684.712764:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4670 00000100:00080000:2.0:1769077684.712770:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077684.712775:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077684.712801:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077684.712805:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616778368:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077684.712824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616778496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077684.712839:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077684.712843:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077684.712846:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077684.712850:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4670 00000100:00080000:2.0:1769077684.712857:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077684.712862:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077684.712868:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077684.712870:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077684.712888:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616778624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077684.712897:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077684.712900:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077684.712902:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077684.712905:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4670 00000100:00080000:2.0:1769077684.712909:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077684.712912:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077684/real 1769077684] req@ffff995b85aca300 x1855007616778368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077684.712915:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077684.712922:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077684.712938:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616778752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077684.712938:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616778368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077684.712954:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077684.712957:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077684.712958:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077684.712964:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4670 00000100:00080000:2.0:1769077684.712966:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077684.712971:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077684.712977:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769077684.712980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077684.712985:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616778880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077684.712991:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077684.712998:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077684.713002:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616778368:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077684.713003:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077684.713006:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077684.713014:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4670 00000100:00080000:2.0:1769077684.713016:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077684.713019:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077684.713022:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077684.713029:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616779008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077684.713036:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077684.713041:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769077684.713041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616778496:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077684.713043:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077684.713051:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4670 00000100:00080000:2.0:1769077684.713054:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077684.713057:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077684.713060:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077684.713069:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616779136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077684.713081:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077684.713100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077684/real 1769077684] req@ffff995b85acaa00 x1855007616778496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077684.713114:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616778496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077684.713129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077684.713133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077684.713137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616778496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077684.713145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616778624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077684.713166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077684.713178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077684/real 1769077684] req@ffff995b85ac8380 x1855007616778624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077684.713191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616778624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077684.713202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077684.713206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077684.713209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616778624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077684.713216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616778752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077684.713235:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077684.713247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077684/real 1769077684] req@ffff995b85acb100 x1855007616778752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077684.713261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616778752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077684.713270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077684.713273:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077684.713277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616778752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077684.713284:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616778880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077684.713303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077684.713315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077684/real 1769077684] req@ffff995b85ac9c00 x1855007616778880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077684.713328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616778880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077684.713338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077684.713341:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077684.713345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616778880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077684.713363:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616779008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077684.713394:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077684.713406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077684/real 1769077684] req@ffff995b85ac9880 x1855007616779008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077684.713417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616779008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077684.713424:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077684.713426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077684.713428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616779008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077684.713432:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616779136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077684.713445:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077684.713453:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077684/real 1769077684] req@ffff995b85acbb80 x1855007616779136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077684.713479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616779136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077684.713485:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077684.713487:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077684.713490:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616779136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077685.416557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077685.416646:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077685.736574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077685.736574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077685.736574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077685.736613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077686.440546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077686.440618:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077686.760536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077686.760551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077686.760558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077686.760570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077687.464565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077687.464620:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077687.784550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077687.784550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077687.784551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077687.784580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077688.488528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077688.488616:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077688.808560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077688.808562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077688.808571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077688.808598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077689.512544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077689.512615:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077689.832539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077689.832540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077689.832540:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077689.832550:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077689.832555:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077689.832564:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4675 00000100:00100000:2.0:1769077689.832569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077689.832575:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077689.832583:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077689.832621:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616779264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077689.832644:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077689.832648:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077689.832649:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077689.832653:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4675 00000100:00080000:3.0:1769077689.832656:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077689.832661:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077689.832669:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077689.832674:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616779264:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077689.832681:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616779392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077689.832690:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077689.832748:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077689.832750:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077689.832754:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077689.832758:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4675 00000100:00080000:3.0:1769077689.832763:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077689.832768:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077689.832773:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077689.832783:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077689.832789:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616779520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077689.832792:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077689/real 1769077689] req@ffff995b84558700 x1855007616779264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077689.832816:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077689.832820:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077689.832822:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077689.832826:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616779264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077689.832827:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4675 00000100:00080000:3.0:1769077689.832832:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077689.832841:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077689.832844:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769077689.832849:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769077689.832850:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077689.832856:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616779264:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077689.832868:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616779648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077689.832879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616779392:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077689.832883:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077689.832889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077689.832891:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077689.832897:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4675 00000800:00000100:2.0:1769077689.832900:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077689.832906:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077689.832912:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077689.832915:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077689/real 1769077689] req@ffff995b8455b480 x1855007616779392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077689.832918:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077689.832929:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616779392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077689.832935:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616779776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077689.832968:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077689.832969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077689.832977:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077689.832977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077689.832982:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077689.832985:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616779392:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077689.832989:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4675 00000100:00080000:3.0:1769077689.832993:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077689.832994:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616779520:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077689.832999:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077689.833004:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077689.833013:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077689.833016:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616779904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077689.833024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077689/real 1769077689] req@ffff995b8455ad80 x1855007616779520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077689.833026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077689.833034:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077689.833038:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077689.833045:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4675 00000100:00100000:2.0:1769077689.833048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616779520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077689.833049:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077689.833056:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077689.833064:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077689.833065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077689.833070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077689.833073:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616780032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077689.833074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616779520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077689.833097:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616779648:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077689.833112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077689.833119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077689/real 1769077689] req@ffff995b8455a300 x1855007616779648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077689.833126:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616779648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077689.833132:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077689.833134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077689.833137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616779648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077689.833141:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616779776:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077689.833149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077689.833155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077689/real 1769077689] req@ffff995b84559c00 x1855007616779776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077689.833162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616779776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077689.833175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077689.833178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077689.833180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616779776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077689.833183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616779904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077689.833194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077689.833204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077689/real 1769077689] req@ffff995b84559880 x1855007616779904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077689.833217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616779904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077689.833233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077689.833237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077689.833241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616779904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077689.833247:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616780032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077689.833268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077689.833282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077689/real 1769077689] req@ffff995b84558000 x1855007616780032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077689.833295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616780032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077689.833301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077689.833303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077689.833306:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616780032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077690.536536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077690.536581:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077690.856568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077690.856575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077690.856575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077690.856610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077691.560511:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077691.560611:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077691.880566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077691.880566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077691.880566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077691.880576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077692.584591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077692.584681:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077692.904570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077692.904577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077692.904581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077692.904615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077693.608548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077693.608603:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077693.928558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077693.928558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077693.928592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077693.928598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077694.632554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077694.632630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077694.952566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077694.952569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077694.952582:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077694.952589:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077694.952593:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077694.952603:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4680 00000100:00100000:1.0:1769077694.952617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077694.952627:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077694.952635:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077694.952648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077694.952676:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616780160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077694.952704:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077694.952708:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077694.952710:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077694.952713:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4680 00000100:00080000:3.0:1769077694.952717:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077694.952721:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077694.952730:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077694.952736:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616780160:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077694.952745:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616780288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077694.952754:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077694.952757:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077694.952759:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077694.952761:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4680 00000100:00080000:3.0:1769077694.952764:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077694.952766:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077694.952770:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077694.952779:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616780416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077694.952785:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077694.952788:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077694.952789:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077694.952792:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4680 00000100:00080000:3.0:1769077694.952795:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077694.952800:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077694.952801:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077694.952806:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077694.952815:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616780544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077694.952822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077694.952824:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077694.952826:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077694.952828:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4680 00000100:00080000:3.0:1769077694.952831:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077694.952833:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077694.952839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077694/real 1769077694] req@ffff995b8455aa00 x1855007616780160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077694.952839:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077694.952852:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616780672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077694.952860:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616780160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077694.952862:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077694.952870:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077694.952874:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077694.952879:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4680 00000100:00080000:2.0:1769077694.952880:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077694.952882:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077694.952887:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077694.952888:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077694.952895:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077694.952897:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616780160:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077694.952906:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616780800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077694.952913:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077694.952916:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077694.952918:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077694.952922:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616780288:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077694.952923:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4680 00000100:00080000:3.0:1769077694.952929:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077694.952932:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077694.952936:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077694.952946:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616780928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077694.952952:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077694.952963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077694/real 1769077694] req@ffff995b84559500 x1855007616780288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077694.952972:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616780288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077694.952979:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077694.952983:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077694.952985:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616780288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077694.952991:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616780416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077694.953002:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077694.953008:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077694/real 1769077694] req@ffff995b8455bb80 x1855007616780416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077694.953016:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616780416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077694.953023:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077694.953025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077694.953027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616780416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077694.953030:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616780544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077694.953041:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077694.953047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077694/real 1769077694] req@ffff995b84558a80 x1855007616780544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077694.953054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616780544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077694.953060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077694.953062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077694.953064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616780544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077694.953068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616780672:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077694.953077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077694.953083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077694/real 1769077694] req@ffff995b8455a680 x1855007616780672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077694.953090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616780672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077694.953096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077694.953098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077694.953099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616780672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077694.953102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616780800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077694.953112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077694.953117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077694/real 1769077694] req@ffff995b8455b100 x1855007616780800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077694.953124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616780800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077694.953130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077694.953133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077694.953135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616780800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077694.953145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616780928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077694.953155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077694.953161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077694/real 1769077694] req@ffff995b8455aa00 x1855007616780928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077694.953168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616780928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077694.953173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077694.953175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077694.953177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616780928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077695.656518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077695.656626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077695.976572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077695.976614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077695.976618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077695.976650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077696.680575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077696.680624:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077697.000591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077697.000593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077697.000601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077697.000617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077697.704519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077697.704614:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077698.024543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077698.024543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077698.024575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077698.024580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077698.728530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077698.728608:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077699.048617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077699.048617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077699.048630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077699.048634:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077699.752553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077699.752606:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077700.072585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077700.072585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077700.072585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077700.072595:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077700.072602:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077700.072608:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077700.072620:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4685 00000100:00080000:3.0:1769077700.072631:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077700.072634:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077700.072638:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077700.072680:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616781056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077700.072706:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077700.072711:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077700.072713:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077700.072718:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4685 00000100:00080000:3.0:1769077700.072722:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077700.072727:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077700.072738:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077700.072739:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616781056:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077700.072751:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616781184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077700.072761:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077700.072764:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077700.072766:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077700.072770:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4685 00000100:00080000:3.0:1769077700.072774:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077700.072778:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077700.072782:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077700.072791:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616781312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077700.072799:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077700.072802:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1769077700.072804:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077700.072804:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077700.072807:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4685 00000100:00080000:3.0:1769077700.072810:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077700.072814:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077700.072818:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077700.072828:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616781440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077700.072837:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077700.072840:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077700.072842:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077700.072845:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4685 00000100:00080000:3.0:1769077700.072848:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077700.072852:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077700/real 1769077700] req@ffff995b84558000 x1855007616781056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077700.072852:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077700.072856:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077700.072865:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616781568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077700.072871:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616781056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077700.072873:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077700.072877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077700.072879:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077700.072882:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077700.072883:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4685 00000100:00080000:3.0:1769077700.072886:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077700.072887:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077700.072890:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077700.072891:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616781056:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769077700.072894:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077700.072903:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616781696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077700.072911:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077700.072914:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077700.072916:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077700.072920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616781184:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077700.072920:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4685 00000100:00080000:3.0:1769077700.072923:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077700.072927:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077700.072932:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077700.072941:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616781824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077700.072942:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077700.072954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077700/real 1769077700] req@ffff995b84559880 x1855007616781184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077700.072968:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616781184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077700.072977:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077700.072980:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077700.072984:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616781184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077700.072991:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616781312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077700.073011:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077700.073020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077700/real 1769077700] req@ffff995b84559c00 x1855007616781312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077700.073044:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616781312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077700.073051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077700.073054:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077700.073056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616781312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077700.073060:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616781440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077700.073072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077700.073079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077700/real 1769077700] req@ffff995b8455a300 x1855007616781440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077700.073086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616781440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077700.073092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077700.073094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077700.073096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616781440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077700.073099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616781568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077700.073108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077700.073113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077700/real 1769077700] req@ffff995b8455ad80 x1855007616781568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077700.073120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616781568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077700.073131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077700.073133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077700.073135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616781568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077700.073138:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616781696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077700.073147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077700.073152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077700/real 1769077700] req@ffff995b8455b480 x1855007616781696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077700.073159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616781696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077700.073164:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077700.073166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077700.073168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616781696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077700.073180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616781824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077700.073189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077700.073194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077700/real 1769077700] req@ffff995b84558700 x1855007616781824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077700.073201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616781824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077700.073206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077700.073208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077700.073211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616781824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077700.776532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077700.776603:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077701.096596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077701.096603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077701.096617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077701.096629:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077701.800523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077701.800646:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077702.120624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077702.120624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077702.120643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077702.120685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077702.824575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077702.824668:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077703.144652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077703.144669:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077703.144701:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077703.144715:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077703.848570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077703.848691:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077704.168532:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077704.168560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077704.168572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077704.168587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077704.872533:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077704.872607:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077705.192549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077705.192556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077705.192560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077705.192585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077705.192595:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077705.192600:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077705.192604:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077705.192612:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4690 00000100:00080000:1.0:1769077705.192623:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077705.192629:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077705.192665:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616781952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077705.192689:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077705.192692:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077705.192693:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077705.192696:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4690 00000100:00080000:1.0:1769077705.192699:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077705.192702:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077705.192711:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077705.192720:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616782080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077705.192728:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616781952:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077705.192731:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077705.192737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077705.192739:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077705.192745:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4690 00000100:00080000:1.0:1769077705.192749:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077705.192753:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077705.192758:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077705.192773:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616782208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077705.192784:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077705.192787:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077705.192789:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769077705.192794:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077705.192797:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4690 00000100:00080000:1.0:1769077705.192802:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077705.192807:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077705.192814:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077705.192830:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616782336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077705.192841:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077705.192843:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077705/real 1769077705] req@ffff995b868c5c00 x1855007616781952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077705.192846:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077705.192850:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077705.192861:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4690 00000100:00080000:1.0:1769077705.192865:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077705.192869:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077705.192874:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077705.192876:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616781952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077705.192886:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616782464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077705.192893:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077705.192896:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077705.192901:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077705.192903:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077705.192906:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077705.192907:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4690 00000100:00080000:1.0:1769077705.192913:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077705.192917:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616781952:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077705.192918:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077705.192924:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077705.192932:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616782592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077705.192938:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077705.192941:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077705.192942:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077705.192945:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4690 00000100:00080000:1.0:1769077705.192947:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077705.192952:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077705.192953:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782080:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769077705.192956:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077705.192968:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616782720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077705.192987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077705.193000:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077705/real 1769077705] req@ffff995b868c6a00 x1855007616782080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077705.193010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616782080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077705.193017:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077705.193019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077705.193022:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077705.193027:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077705.193038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077705.193045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077705/real 1769077705] req@ffff995b868c7100 x1855007616782208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077705.193053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616782208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077705.193060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077705.193062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077705.193070:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077705.193075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782336:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077705.193087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077705.193093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077705/real 1769077705] req@ffff995b868c7480 x1855007616782336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077705.193101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616782336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077705.193111:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077705.193113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077705.193115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077705.193119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782464:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077705.193128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077705.193133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077705/real 1769077705] req@ffff995b868c7800 x1855007616782464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077705.193140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616782464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077705.193147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077705.193149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077705.193150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077705.193154:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616782592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077705.193163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077705.193168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077705/real 1769077705] req@ffff995b868c5500 x1855007616782592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077705.193176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616782592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077705.193181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077705.193182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077705.193185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616782592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077705.193198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616782720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077705.193215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077705.193226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077705/real 1769077705] req@ffff995b868c4700 x1855007616782720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077705.193239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616782720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077705.193249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077705.193252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077705.193256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616782720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077705.896582:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077705.896663:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077706.216538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077706.216544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077706.216550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077706.216562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077706.920556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077706.920615:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077707.240563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077707.240563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077707.240563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077707.240564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077707.944551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077707.944619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077708.264522:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077708.264538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077708.264554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077708.264588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077708.968527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077708.968573:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077709.288568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077709.288569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077709.288569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077709.288576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077709.992567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077709.992630:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077710.312535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077710.312563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077710.312572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077710.312578:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077710.312584:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077710.312589:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077710.312591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077710.312600:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4696 00000100:00080000:0.0:1769077710.312610:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077710.312617:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077710.312652:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616782848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077710.312675:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077710.312679:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077710.312681:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077710.312685:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4696 00000100:00080000:0.0:1769077710.312690:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077710.312695:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077710.312704:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077710.312718:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616782976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077710.312720:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616782848:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077710.312728:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077710.312731:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077710.312732:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077710.312736:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4696 00000100:00080000:0.0:1769077710.312740:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077710.312743:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077710.312747:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077710.312757:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616783104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077710.312765:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077710.312768:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077710.312769:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077710.312773:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4696 00000100:00080000:0.0:1769077710.312777:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077710.312781:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077710.312786:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077710.312798:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616783232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077710.312807:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077710.312810:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077710.312812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077710.312815:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4696 00000100:00080000:0.0:1769077710.312818:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077710.312821:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077710.312826:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077710.312834:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616783360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077710.312835:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077710.312841:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077710.312844:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077710.312846:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077710.312849:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4696 00000100:00080000:0.0:1769077710.312851:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077710.312854:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077710.312858:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077710.312866:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616783488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077710.312867:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077710/real 1769077710] req@ffff995b84641500 x1855007616782848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077710.312875:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077710.312878:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077710.312880:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077710.312883:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4696 00000100:00080000:0.0:1769077710.312886:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077710.312889:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077710.312894:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077710.312904:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616783616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077710.312977:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616782848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077710.312987:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077710.312991:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077710.312994:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616782848:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077710.313053:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077710.313076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077710.313088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077710/real 1769077710] req@ffff995b84641180 x1855007616782976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077710.313096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616782976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077710.313102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077710.313104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077710.313106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616782976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077710.313109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616783104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077710.313121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077710.313127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077710/real 1769077710] req@ffff995b84643800 x1855007616783104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077710.313138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616783104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077710.313143:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077710.313145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077710.313146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616783104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077710.313149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616783232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077710.313159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077710.313164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077710/real 1769077710] req@ffff995b84643480 x1855007616783232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077710.313170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616783232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077710.313176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077710.313177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077710.313179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616783232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077710.313181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616783360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077710.313190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077710.313194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077710/real 1769077710] req@ffff995b84642d80 x1855007616783360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077710.313200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616783360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077710.313209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077710.313211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077710.313212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616783360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077710.313215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616783488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077710.313223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077710.313228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077710/real 1769077710] req@ffff995b84643100 x1855007616783488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077710.313234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616783488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077710.313238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077710.313240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077710.313241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616783488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077710.313244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616783616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077710.313252:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077710.313256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077710/real 1769077710] req@ffff995b84643b80 x1855007616783616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077710.313262:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616783616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077710.313266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077710.313268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077710.313270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616783616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077711.016510:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077711.016579:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077711.336576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077711.336607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077711.336633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077711.336634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077712.040552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077712.040637:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077712.360614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077712.360622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077712.360650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077712.360672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077713.064523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077713.064582:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077713.384553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077713.384553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077713.384565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077713.384575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077714.088544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077714.088601:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077714.408561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077714.408561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077714.408561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077714.408562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077715.112545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077715.112598:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077715.432539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077715.432541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077715.432570:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077715.432575:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077715.432579:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077715.432587:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4701 00000100:00080000:0.0:1769077715.432608:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077715.432613:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077715.432644:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616783744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077715.432664:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077715.432667:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077715.432668:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077715.432671:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4701 00000100:00080000:0.0:1769077715.432673:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077715.432675:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077715.432693:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077715.432700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616783744:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077715.432704:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616783872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077715.432712:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077715.432715:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077715.432717:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077715.432720:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4701 00000100:00080000:0.0:1769077715.432722:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077715.432724:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077715.432728:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077715.432735:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616784000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077715.432740:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077715.432742:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077715.432743:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077715.432745:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4701 00000100:00080000:0.0:1769077715.432747:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077715.432748:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077715.432751:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077715.432757:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616784128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077715.432762:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077715.432764:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077715.432765:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077715.432767:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4701 00000100:00080000:0.0:1769077715.432769:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077715.432773:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769077715.432775:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077715.432776:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077715.432784:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616784256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077715.432790:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077715.432792:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077715.432793:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077715.432795:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4701 00000100:00080000:0.0:1769077715.432798:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077715.432800:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077715.432802:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077715.432808:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616784384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077715.432813:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077715.432815:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077715.432817:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077715.432819:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4701 00000100:00080000:0.0:1769077715.432821:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077715.432825:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077715.432826:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077715/real 1769077715] req@ffff995b84642a00 x1855007616783744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077715.432829:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077715.432839:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616784512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077715.432857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616783744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077715.432869:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077715.432874:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077715.432877:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616783744:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077715.432901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616783872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077715.432926:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077715.432939:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077715/real 1769077715] req@ffff995b84640700 x1855007616783872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077715.432955:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077715.432959:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616783872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077715.432970:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077715.432974:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077715.432977:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616783872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077715.432982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077715.433001:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000800:00000100:2.0:1769077715.433009:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077715.433022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077715/real 1769077715] req@ffff995b84641880 x1855007616784000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077715.433034:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616784000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077715.433053:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077715.433055:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077715.433058:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077715.433062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784128:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077715.433076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077715.433085:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077715/real 1769077715] req@ffff995b84643b80 x1855007616784128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077715.433093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616784128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077715.433098:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077715.433100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077715.433102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077715.433106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077715.433116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077715.433123:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077715/real 1769077715] req@ffff995b84643100 x1855007616784256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077715.433140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616784256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077715.433148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077715.433152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077715.433155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077715.433160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616784384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077715.433179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077715.433189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077715/real 1769077715] req@ffff995b84642d80 x1855007616784384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077715.433200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616784384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077715.433207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077715.433210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077715.433213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616784384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077715.433218:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616784512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077715.433236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077715.433243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077715/real 1769077715] req@ffff995b84643480 x1855007616784512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077715.433249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616784512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077715.433258:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077715.433260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077715.433276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616784512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077716.136598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077716.136665:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077716.456566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077716.456566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077716.456566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077716.456566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077717.160552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077717.160609:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077717.480565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077717.480566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077717.480566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077717.480576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077718.184561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077718.184618:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077718.504567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077718.504567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077718.504567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077718.504567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077719.208523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077719.208579:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077719.528549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077719.528549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077719.528551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077719.529062:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077720.232562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077720.232616:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077720.552603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077720.552603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077720.552608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077720.552620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077720.552675:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077720.552680:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077720.552684:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077720.552694:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4706 00000100:00080000:2.0:1769077720.552704:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077720.552712:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077720.552751:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616784640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077720.552786:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077720.552795:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077720.552796:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077720.552802:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4706 00000100:00080000:2.0:1769077720.552806:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077720.552810:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077720.552817:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077720.552828:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616784768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077720.552836:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616784640:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077720.552840:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077720.552850:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077720.552852:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077720.552855:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4706 00000100:00080000:2.0:1769077720.552857:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077720.552859:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077720.552862:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077720.552872:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616784896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077720.552879:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077720.552882:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077720.552883:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077720.552886:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4706 00000100:00080000:2.0:1769077720.552887:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077720.552889:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077720.552892:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077720.552900:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616785024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077720.552906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077720.552908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077720.552910:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077720.552913:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4706 00000100:00080000:2.0:1769077720.552915:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077720.552916:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077720.552919:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077720.552931:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616785152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077720.552932:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077720.552942:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077720.552944:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077720.552945:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077720.552948:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4706 00000100:00080000:2.0:1769077720.552952:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077720.552955:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077720.552957:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077720.552966:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616785280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077720.552972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077720.552974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077720.552975:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077720.552978:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4706 00000100:00080000:2.0:1769077720.552982:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077720.552987:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077720.552990:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077720.552995:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077720/real 1769077720] req@ffff995b85ac8a80 x1855007616784640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077720.553001:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616785408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077720.553052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616784640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077720.553066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077720.553071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077720.553075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616784640:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077720.553100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077720.553121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077720.553132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077720/real 1769077720] req@ffff995b85acb800 x1855007616784768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077720.553151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616784768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077720.553159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077720.553161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077720.553165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077720.553170:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077720.553200:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077720.553210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077720/real 1769077720] req@ffff995b85acbb80 x1855007616784896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077720.553222:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616784896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077720.553232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077720.553235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077720.553238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616784896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077720.553244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785024:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077720.553259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077720.553267:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077720/real 1769077720] req@ffff995b85ac9880 x1855007616785024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077720.553278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616785024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077720.553292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077720.553295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077720.553298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077720.553303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785152:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077720.553317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077720.553325:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077720/real 1769077720] req@ffff995b85ac9c00 x1855007616785152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077720.553336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616785152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077720.553344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077720.553347:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077720.553350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077720.553356:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616785280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077720.553370:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077720.553377:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077720/real 1769077720] req@ffff995b85acb100 x1855007616785280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077720.553388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616785280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077720.553396:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077720.553398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077720.553409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616785280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077720.553415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616785408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077720.553434:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077720.553444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077720/real 1769077720] req@ffff995b85ac8380 x1855007616785408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077720.553485:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616785408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077720.553494:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077720.553497:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077720.553499:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616785408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077721.256548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077721.256600:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077721.576533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077721.576539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077721.576640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077721.576641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077722.280529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077722.280588:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077722.600568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077722.600568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077722.600574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077722.600582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077723.304547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077723.304600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077723.624537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077723.624544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077723.624575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077723.624577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077724.328529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077724.328584:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077724.648559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077724.648559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077724.648593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077724.648605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077725.352562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077725.352611:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077725.672546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077725.672546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077725.672564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077725.672574:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077725.672582:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077725.672588:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077725.672592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077725.672599:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4711 00000100:00080000:0.0:1769077725.672611:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077725.672619:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077725.672658:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616785536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077725.672689:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077725.672693:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077725.672694:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077725.672699:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4711 00000100:00080000:0.0:1769077725.672702:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077725.672707:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077725.672716:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077725.672730:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616785664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077725.672738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616785536:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077725.672743:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077725.672748:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077725.672750:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077725.672754:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4711 00000100:00080000:0.0:1769077725.672765:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077725.672769:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077725.672773:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077725.672786:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616785792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077725.672796:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077725.672799:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077725.672801:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077725.672804:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4711 00000800:00000100:3.0:1769077725.672809:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077725.672810:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077725.672817:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077725.672822:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077725.672833:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616785920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077725.672843:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077725.672846:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077725.672848:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077725.672852:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4711 00000100:00100000:3.0:1769077725.672857:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077725/real 1769077725] req@ffff995b84643800 x1855007616785536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077725.672858:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077725.672865:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077725.672872:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077725.672884:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616786048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077725.672893:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077725.672894:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616785536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077725.672899:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077725.672902:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077725.672912:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4711 00000100:00080000:0.0:1769077725.672917:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077725.672925:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077725.672929:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769077725.672930:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077725.672937:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077725.672944:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616785536:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077725.672945:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616786176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077725.672958:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077725.672962:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077725.672964:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077725.672967:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4711 00000100:00080000:0.0:1769077725.672971:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077725.672975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785664:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077725.672978:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077725.672986:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077725.672998:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616786304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077725.673010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077725.673030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077725/real 1769077725] req@ffff995b84641180 x1855007616785664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077725.673046:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616785664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077725.673054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077725.673058:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077725.673061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077725.673067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077725.673085:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077725.673124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077725/real 1769077725] req@ffff995b84641500 x1855007616785792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077725.673136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616785792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077725.673146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077725.673150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077725.673153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077725.673159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077725.673189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077725.673200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077725/real 1769077725] req@ffff995b84642680 x1855007616785920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077725.673211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616785920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077725.673219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077725.673222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077725.673225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616785920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077725.673230:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077725.673249:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077725.673259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077725/real 1769077725] req@ffff995b84640e00 x1855007616786048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077725.673270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616786048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077725.673279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077725.673282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077725.673285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077725.673291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616786176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077725.673306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077725.673315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077725/real 1769077725] req@ffff995b84641c00 x1855007616786176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077725.673327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616786176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077725.673335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077725.673337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077725.673340:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616786176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077725.673358:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616786304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077725.673386:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077725.673401:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077725/real 1769077725] req@ffff995b84640a80 x1855007616786304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077725.673413:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616786304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077725.673421:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077725.673424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077725.673427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616786304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077726.376552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077726.376613:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077726.696565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077726.696565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077726.696565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077726.696583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077727.400570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077727.400632:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077727.720537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077727.720537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077727.720537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077727.720542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077728.424543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077728.424616:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077728.744564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077728.744564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077728.744679:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077728.744744:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077729.448555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077729.448615:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077729.768531:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077729.768547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077729.768548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077729.768555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077730.472523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077730.472565:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077730.792550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077730.792550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077730.792551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077730.792553:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077730.792563:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077730.792568:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077730.792577:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4716 00000100:00080000:0.0:1769077730.792589:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077730.792592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077730.792598:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077730.792638:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616786432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077730.792666:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077730.792670:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077730.792673:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077730.792678:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4716 00000100:00080000:0.0:1769077730.792682:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077730.792686:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077730.792695:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077730.792705:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616786432:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077730.792710:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616786560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077730.792722:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077730.792726:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077730.792729:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077730.792733:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4716 00000100:00080000:0.0:1769077730.792737:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077730.792741:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077730.792746:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077730.792757:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616786688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077730.792767:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077730.792770:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077730.792772:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077730.792776:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4716 00000100:00080000:0.0:1769077730.792781:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077730.792785:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077730.792792:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077730.792807:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077730.792816:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616786816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077730.792828:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077730.792832:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077730.792834:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077730.792838:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4716 00000100:00080000:0.0:1769077730.792843:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077730.792846:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786688:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077730.792850:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077730.792858:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077730.792873:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616786944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077730.792878:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786816:192.168.204.132@tcp:8:kworker.0 00000100:00080000:0.0:1769077730.792890:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077730.792894:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077730.792896:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077730.792901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786944:192.168.204.132@tcp:8:kworker.0 00000100:00080000:0.0:1769077730.792905:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4716 00000100:00080000:0.0:1769077730.792911:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077730.792916:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077730.792921:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077730.792934:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616787072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077730.792946:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077730.792950:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077730.792952:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077730.792956:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4716 00000100:00080000:0.0:1769077730.792961:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077730.792967:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077730.792968:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616787072:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769077730.792974:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077730.792988:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616787200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077730.793015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616787200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077731.496559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077731.496625:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077731.816614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077731.816636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077731.816657:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077731.816691:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077732.520516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077732.520574:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077732.840578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077732.840579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077732.840581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077732.840629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077733.544573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077733.544639:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077733.864567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077733.864567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077733.864575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077733.864703:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000400:00000100:2.0:1769077733.864778:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769077733.864822:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077733.864843:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077733.864896:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077733.864907:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077733.864916:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077733.864924:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077733.864932:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077733.864940:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769077733.865088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077730/real 1769077733] req@ffff995b84642680 x1855007616787200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077733.865118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616787200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077733.865131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077733.865137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077733.865141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616787200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077733.865149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077730/real 1769077733] req@ffff995b84640e00 x1855007616787072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077733.865162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616787072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077733.865171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077733.865175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077733.865179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616787072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077733.865186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077730/real 1769077733] req@ffff995b84641c00 x1855007616786944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077733.865200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616786944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077733.865210:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077733.865214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077733.865217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077733.865224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077730/real 1769077733] req@ffff995b84640a80 x1855007616786816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077733.865236:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616786816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077733.865245:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077733.865248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077733.865252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077733.865259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077730/real 1769077733] req@ffff995b84641f80 x1855007616786560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077733.865270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616786560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077733.865278:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077733.865282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077733.865286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077733.865293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077730/real 1769077733] req@ffff995b84640380 x1855007616786688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077733.865305:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616786688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077733.865316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077733.865319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077733.865323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616786688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077733.865331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077730/real 1769077733] req@ffff995b84640000 x1855007616786432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077733.865343:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616786432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077733.865351:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077733.865355:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077733.865358:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616786432:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077734.568578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077734.568608:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077734.888574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077734.888606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077734.888614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077734.888660:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077735.592556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077735.592613:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077735.912560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077735.912560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077735.912599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077735.912612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077735.912631:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077735.912638:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077735.912643:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077735.912654:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4721 00000100:00080000:3.0:1769077735.912666:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077735.912672:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077735.912712:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616787328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077735.912751:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077735.912756:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077735.912758:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077735.912768:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4721 00000100:00080000:3.0:1769077735.912773:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077735.912777:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077735.912788:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077735.912803:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616787456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077735.912814:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077735.912817:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077735.912819:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077735.912823:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4721 00000100:00080000:3.0:1769077735.912827:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077735.912831:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077735.912836:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077735.912847:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616787584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077735.912857:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077735.912860:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077735.912862:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077735.912867:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4721 00000100:00080000:3.0:1769077735.912871:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077735.912875:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077735.912882:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077735.912896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616787328:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077735.912897:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616787712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077735.912914:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077735.912930:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077735.912932:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077735.912937:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4721 00000100:00080000:3.0:1769077735.912941:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077735.912945:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077735.912951:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077735.912965:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616787840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077735.912975:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077735.912979:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077735.912982:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077735.912986:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4721 00000100:00080000:3.0:1769077735.912990:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077735.912994:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769077735.913001:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769077735.913001:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077735.913018:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616787968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077735.913028:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077735.913031:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077735.913033:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077735.913038:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4721 00000100:00080000:3.0:1769077735.913043:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077735.913048:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077735.913055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077735/real 1769077735] req@ffff995b84558700 x1855007616787328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077735.913055:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077735.913072:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616788096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077735.913084:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616787328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077735.913099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077735.913104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077735.913109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616787328:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077735.913119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616787456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077735.913824:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077735.913849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077735/real 1769077735] req@ffff995b8455b480 x1855007616787456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077735.913867:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616787456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077735.913877:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077735.913882:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077735.913886:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616787456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077735.913895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616787584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077735.913951:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077735.913966:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077735/real 1769077735] req@ffff995b8455ad80 x1855007616787584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077735.913978:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616787584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077735.913988:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077735.913991:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077735.913995:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616787584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077735.914027:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616787712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077735.914050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077735.914061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077735/real 1769077735] req@ffff995b8455a300 x1855007616787712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077735.914074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616787712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077735.914090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077735.914094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077735.914097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616787712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077735.914104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616787840:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077735.914125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077735.914136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077735/real 1769077735] req@ffff995b84559c00 x1855007616787840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077735.914162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616787840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077735.914173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077735.914177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077735.914180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616787840:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077735.914188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616787968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077735.914209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077735.914221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077735/real 1769077735] req@ffff995b84559880 x1855007616787968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077735.914234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616787968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077735.914247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077735.914250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077735.914254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616787968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077735.914261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616788096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077735.914282:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077735.914295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077735/real 1769077735] req@ffff995b84558000 x1855007616788096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077735.914308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616788096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077735.914317:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077735.914321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077735.914325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616788096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077736.616558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077736.616649:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077736.936562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077736.936562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077736.936663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077736.936681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077737.640542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077737.640606:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077737.960567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077737.960572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077737.960581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077737.960603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077738.664572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077738.664688:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077738.984544:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077738.984568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077738.984604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077738.984642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077739.688553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077739.688680:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077740.008598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077740.008608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077740.008609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077740.008632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077740.712569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077740.712677:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077741.032570:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077741.032571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077741.032576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077741.032583:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077741.032590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077741.032601:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4726 00000100:00100000:2.0:1769077741.032604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077741.032614:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077741.032617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077741.032626:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077741.032667:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616788224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077741.032687:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077741.032698:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077741.032701:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077741.032704:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4726 00000100:00080000:3.0:1769077741.032707:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077741.032712:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077741.032721:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077741.032728:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616788224:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077741.032737:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616788352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077741.032745:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077741.032748:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077741.032750:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077741.032754:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4726 00000100:00080000:3.0:1769077741.032757:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077741.032760:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077741.032763:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077741.032770:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616788480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077741.032777:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077741.032780:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077741.032782:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077741.032786:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4726 00000100:00080000:3.0:1769077741.032795:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077741.032799:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077741.032803:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077741.032814:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616788608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077741.032821:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077741.032829:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077741.032854:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077741.032857:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077741.032863:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4726 00000100:00080000:3.0:1769077741.032867:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077741.032871:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077741.032875:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077741.032887:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616788736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077741.032895:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077741.032898:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077741.032899:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077741.032905:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077741/real 1769077741] req@ffff995b8455aa00 x1855007616788224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077741.032907:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4726 00000100:00080000:3.0:1769077741.032913:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077741.032920:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077741.032932:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077741.032948:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616788864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077741.032964:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077741.032967:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077741.032969:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077741.032975:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4726 00000100:00100000:2.0:1769077741.032977:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616788224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077741.032979:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077741.032989:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077741.032995:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077741.033013:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616788992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077741.033014:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077741.033030:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077741.033039:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616788224:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077741.033077:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616788352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077741.033102:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077741.033115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077741/real 1769077741] req@ffff995b8455b100 x1855007616788352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077741.033128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616788352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077741.033137:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077741.033141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077741.033145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616788352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077741.033151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616788480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077741.033169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077741.033179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077741/real 1769077741] req@ffff995b8455a680 x1855007616788480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077741.033191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616788480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077741.033208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077741.033212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077741.033215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616788480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077741.033222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616788608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077741.033240:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077741.033250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077741/real 1769077741] req@ffff995b84558a80 x1855007616788608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077741.033261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616788608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077741.033271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077741.033274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077741.033278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616788608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077741.033284:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616788736:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077741.033302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077741.033313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077741/real 1769077741] req@ffff995b8455bb80 x1855007616788736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077741.033324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616788736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077741.033333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077741.033336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077741.033340:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616788736:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077741.033346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616788864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077741.033363:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077741.033372:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077741/real 1769077741] req@ffff995b84559500 x1855007616788864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077741.033382:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616788864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077741.033390:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077741.033393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077741.033396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616788864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077741.033401:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616788992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077741.033416:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077741.033424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077741/real 1769077741] req@ffff995b84558000 x1855007616788992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077741.033434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616788992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077741.033442:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077741.033445:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077741.033448:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616788992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077741.736559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077741.736617:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077742.056587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077742.056588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077742.056611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077742.056616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077742.760577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077742.760647:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077743.080524:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077743.080533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077743.080542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077743.080558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077743.784562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077743.784632:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077744.104535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077744.104580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077744.104595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077744.104629:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077744.808525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077744.808705:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077745.128523:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077745.128540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077745.128573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077745.128589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077745.832529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077745.832645:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077746.152521:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077746.152521:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077746.152538:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077746.152544:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077746.152549:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077746.152559:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4731 00000100:00100000:0.0:1769077746.152564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077746.152571:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077746.152581:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077746.152595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077746.152623:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616789120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077746.152649:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077746.152652:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077746.152654:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077746.152657:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4731 00000100:00080000:2.0:1769077746.152663:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077746.152668:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077746.152679:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077746.152689:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616789120:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077746.152697:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616789248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077746.152713:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077746.152715:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077746.152717:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077746.152722:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4731 00000100:00080000:2.0:1769077746.152734:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077746.152738:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077746.152743:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077746.152769:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616789376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077746.152779:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077746.152780:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077746.152786:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077746.152787:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077746.152796:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4731 00000100:00080000:2.0:1769077746.152801:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077746.152804:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077746.152810:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077746.152824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616789504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077746.152835:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077746.152838:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077746/real 1769077746] req@ffff995b85acaa00 x1855007616789120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077746.152840:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077746.152845:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077746.152855:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4731 00000100:00080000:2.0:1769077746.152860:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077746.152863:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616789120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077746.152866:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077746.152874:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769077746.152878:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077746.152884:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077746.152888:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616789632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077746.152891:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616789120:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077746.152910:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077746.152913:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077746.152915:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077746.152919:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4731 00000100:00080000:2.0:1769077746.152922:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077746.152925:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077746.152930:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616789248:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077746.152931:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077746.152943:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616789760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077746.152952:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077746.152955:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077746.152956:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077746.152959:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4731 00000100:00080000:2.0:1769077746.152961:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077746.152965:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077746.152968:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077746.152970:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077746.152981:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616789888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077746.152989:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077746/real 1769077746] req@ffff995b85aca300 x1855007616789248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077746.153008:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616789248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077746.153018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077746.153023:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077746.153027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616789248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077746.153035:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616789376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077746.153072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077746.153088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077746/real 1769077746] req@ffff995b85ac8380 x1855007616789376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077746.153103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616789376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077746.153115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077746.153119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077746.153122:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616789376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077746.153130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616789504:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077746.153159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077746.153178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077746/real 1769077746] req@ffff995b85acb100 x1855007616789504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077746.153193:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616789504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077746.153203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077746.153205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077746.153208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616789504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077746.153212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616789632:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077746.153228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077746.153236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077746/real 1769077746] req@ffff995b85ac9c00 x1855007616789632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077746.153244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616789632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077746.153250:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077746.153252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077746.153254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616789632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077746.153267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616789760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077746.153278:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077746.153284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077746/real 1769077746] req@ffff995b85ac9880 x1855007616789760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077746.153292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616789760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077746.153302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077746.153304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077746.153306:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616789760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077746.153310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616789888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077746.153318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077746.153324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077746/real 1769077746] req@ffff995b85acbb80 x1855007616789888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077746.153335:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616789888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077746.153344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077746.153348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077746.153351:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616789888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077746.856555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077746.856641:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077747.176525:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077747.176544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077747.176549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077747.176576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077747.880555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077747.880614:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077748.200539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077748.200539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077748.200539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077748.201493:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769077748.904626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077748.904669:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077749.224529:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077749.224545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077749.224559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077749.224586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077749.928534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077749.928593:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077750.248550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077750.248555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077750.248563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077750.248567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077750.952557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077750.952612:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077751.272526:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077751.272531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077751.272534:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077751.272648:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077751.272654:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077751.272658:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077751.272700:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4736 00000100:00100000:3.0:1769077751.272701:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077751.272714:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077751.272723:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077751.272771:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616790016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077751.272794:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077751.272797:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077751.272798:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077751.272802:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4736 00000100:00080000:0.0:1769077751.272805:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077751.272809:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077751.272818:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077751.272823:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616790016:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077751.272830:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616790144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077751.272839:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077751.272841:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077751.272842:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077751.272845:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4736 00000100:00080000:0.0:1769077751.272847:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077751.272850:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077751.272853:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077751.272862:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616790272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077751.272868:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077751.272871:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077751.272872:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077751.272875:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4736 00000100:00080000:0.0:1769077751.272879:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077751.272881:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077751.272884:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077751.272889:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077751.272893:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616790400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077751.272901:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077751.272904:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077751.272905:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077751.272908:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4736 00000100:00080000:0.0:1769077751.272910:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077751.272912:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077751.272915:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077751.272924:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616790528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077751.272928:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077751/real 1769077751] req@ffff995b84641500 x1855007616790016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077751.272934:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077751.272952:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077751.272953:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077751.272958:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4736 00000100:00100000:2.0:1769077751.272960:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616790016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077751.272962:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077751.272969:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077751.272977:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077751.272978:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077751.272984:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077751.272987:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616790656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077751.272991:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616790016:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077751.273003:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077751.273006:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077751.273007:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077751.273010:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4736 00000100:00080000:0.0:1769077751.273012:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077751.273015:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077751.273018:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077751.273019:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616790144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077751.273028:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616790784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077751.273043:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077751.273054:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077751/real 1769077751] req@ffff995b84641180 x1855007616790144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077751.273063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616790144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077751.273069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077751.273071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077751.273073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616790144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077751.273080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616790272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077751.273091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077751.273097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077751/real 1769077751] req@ffff995b84643800 x1855007616790272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077751.273105:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616790272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077751.273112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077751.273114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077751.273116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616790272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077751.273119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616790400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077751.273130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077751.273136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077751/real 1769077751] req@ffff995b84643480 x1855007616790400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077751.273143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616790400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077751.273149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077751.273151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077751.273153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616790400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077751.273157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616790528:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077751.273166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077751.273172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077751/real 1769077751] req@ffff995b84642d80 x1855007616790528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077751.273179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616790528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077751.273185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077751.273187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077751.273189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616790528:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077751.273192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616790656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077751.273201:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077751.273207:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077751/real 1769077751] req@ffff995b84643100 x1855007616790656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077751.273214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616790656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077751.273220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077751.273222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077751.273226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616790656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077751.273254:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616790784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077751.273277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077751.273289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077751/real 1769077751] req@ffff995b84643b80 x1855007616790784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077751.273298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616790784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077751.273308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077751.273311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077751.273313:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616790784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077751.976532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077751.976616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077752.296614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077752.296617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077752.296665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077752.296692:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077753.000565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077753.000658:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077753.320575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077753.320575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077753.320575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077753.320647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077754.024567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077754.024651:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077754.344597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077754.344602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077754.344624:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077754.344647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077755.048581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077755.048689:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077755.368614:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077755.368628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077755.368649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077755.368682:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077756.072565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077756.072654:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077756.392611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077756.392611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077756.392611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077756.392639:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077756.392671:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077756.392679:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077756.392684:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077756.392695:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4742 00000100:00080000:1.0:1769077756.392707:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077756.392715:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077756.392756:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616790912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077756.392783:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077756.392786:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077756.392787:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077756.392790:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4742 00000100:00080000:1.0:1769077756.392793:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077756.392797:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077756.392804:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077756.392814:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616791040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077756.392821:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077756.392826:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077756.392828:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077756.392828:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616790912:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077756.392836:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4742 00000100:00080000:1.0:1769077756.392840:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077756.392842:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077756.392845:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077756.392853:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616791168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077756.392859:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077756.392861:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077756.392863:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077756.392865:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4742 00000100:00080000:1.0:1769077756.392868:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077756.392870:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077756.392874:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077756.392882:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616791296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077756.392887:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077756.392889:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077756.392891:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077756.392893:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4742 00000100:00080000:1.0:1769077756.392895:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077756.392897:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077756.392900:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077756.392906:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616791424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077756.392911:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077756.392913:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077756.392914:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077756.392918:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4742 00000100:00080000:1.0:1769077756.392923:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077756.392924:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077756.392928:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077756.392933:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077756.392939:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616791552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077756.392951:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077756.392954:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077756.392955:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077756.392958:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4742 00000100:00080000:1.0:1769077756.392960:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077756.392962:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077756.392965:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077756.392970:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616791680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077756.392988:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077756/real 1769077756] req@ffff995b868c4a80 x1855007616790912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077756.393011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616790912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077756.393024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077756.393029:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077756.393034:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616790912:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077756.393059:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077756.393086:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077756.393101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077756/real 1769077756] req@ffff995b868c6d80 x1855007616791040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077756.393115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616791040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077756.393124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077756.393128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077756.393132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077756.393139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077756.393160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077756.393172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077756/real 1769077756] req@ffff995b868c6300 x1855007616791168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077756.393185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616791168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077756.393196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077756.393200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077756.393204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077756.393211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791296:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077756.393232:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077756.393244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077756/real 1769077756] req@ffff995b868c4700 x1855007616791296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077756.393257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616791296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077756.393268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077756.393272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077756.393275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077756.393282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791424:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077756.393303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077756.393315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077756/real 1769077756] req@ffff995b868c5500 x1855007616791424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077756.393328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616791424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077756.393339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077756.393343:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077756.393346:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791424:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077756.393353:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616791552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077756.393374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077756.393386:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077756/real 1769077756] req@ffff995b868c7800 x1855007616791552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077756.393400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616791552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077756.393410:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077756.393413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077756.393418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616791552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077756.393424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616791680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077756.393444:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077756.393479:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077756/real 1769077756] req@ffff995b868c7480 x1855007616791680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077756.393495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616791680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077756.393504:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077756.393508:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077756.393512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616791680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077757.096576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077757.096683:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077757.416599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077757.416599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077757.416615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077757.416645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077758.120586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077758.120627:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077758.440613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077758.440613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077758.440613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077758.440679:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077759.144587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077759.144651:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077759.464615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077759.464615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077759.464656:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077759.464671:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077760.168581:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077760.168583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077760.488564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077760.488571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077760.488670:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077760.488671:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077761.192575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077761.192640:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077761.512582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077761.512597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077761.512599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077761.512609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077761.512636:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077761.512642:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077761.512646:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077761.512655:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4747 00000100:00080000:0.0:1769077761.512666:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077761.512673:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077761.512717:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616791808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077761.512752:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077761.512755:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077761.512757:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077761.512761:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4747 00000100:00080000:0.0:1769077761.512764:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077761.512768:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077761.512778:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077761.512791:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616791936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077761.512795:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616791808:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077761.512803:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077761.512807:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077761.512809:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077761.512811:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4747 00000100:00080000:0.0:1769077761.512814:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077761.512816:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077761.512819:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077761.512828:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616792064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077761.512836:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077761.512839:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077761.512840:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077761.512843:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4747 00000100:00080000:0.0:1769077761.512845:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077761.512848:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077761.512851:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077761.512858:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616792192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077761.512865:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077761.512867:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077761.512869:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769077761.512873:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077761.512874:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4747 00000100:00080000:0.0:1769077761.512879:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077761.512882:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077761.512885:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077761.512894:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616792320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077761.512901:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077761.512904:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077761.512905:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077761.512909:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4747 00000100:00080000:0.0:1769077761.512913:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077761.512916:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077761/real 1769077761] req@ffff995b84641880 x1855007616791808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077761.512919:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077761.512926:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077761.512937:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616792448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077761.512944:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077761.512948:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616791808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077761.512949:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077761.512952:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077761.512960:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4747 00000100:00080000:0.0:1769077761.512964:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077761.512965:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077761.512968:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077761.512972:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077761.512972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077761.512978:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616791808:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077761.512983:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616792576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077761.513005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077761.513023:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077761.513032:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077761/real 1769077761] req@ffff995b84640700 x1855007616791936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077761.513040:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616791936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077761.513046:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077761.513048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077761.513050:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616791936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077761.513054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077761.513065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077761.513071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077761/real 1769077761] req@ffff995b84642a00 x1855007616792064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077761.513078:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616792064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077761.513091:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077761.513093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077761.513095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077761.513098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077761.513109:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077761.513115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077761/real 1769077761] req@ffff995b84643b80 x1855007616792192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077761.513122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616792192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077761.513129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077761.513131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077761.513133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077761.513136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792320:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077761.513146:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077761.513151:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077761/real 1769077761] req@ffff995b84643100 x1855007616792320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077761.513159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616792320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077761.513166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077761.513168:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077761.513172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792320:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077761.513177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616792448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077761.513197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077761.513208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077761/real 1769077761] req@ffff995b84642d80 x1855007616792448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077761.513221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616792448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077761.513231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077761.513234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077761.513237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616792448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077761.513241:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616792576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077761.513263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077761.513275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077761/real 1769077761] req@ffff995b84643480 x1855007616792576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077761.513289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616792576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077761.513298:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077761.513301:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077761.513305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616792576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077762.216557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077762.216581:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077762.536569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077762.536569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077762.536569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077762.536618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769077763.240644:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077763.240694:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077763.560572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077763.560572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077763.560572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077763.560591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077764.264544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077764.264609:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077764.584555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077764.584555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077764.584555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077764.584567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077765.288556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077765.288637:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077765.608572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077765.608572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077765.608670:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077765.608674:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077766.312558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077766.312625:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077766.632560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077766.632563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077766.632572:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077766.632579:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077766.632583:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077766.632585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077766.632593:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4752 00000100:00100000:3.0:1769077766.632595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077766.632606:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077766.632614:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077766.632651:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616792704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077766.632676:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077766.632678:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077766.632680:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077766.632683:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4752 00000100:00080000:0.0:1769077766.632685:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077766.632687:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077766.632695:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077766.632704:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616792832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077766.632712:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077766.632715:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077766.632718:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077766.632719:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616792704:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077766.632723:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4752 00000100:00080000:0.0:1769077766.632729:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077766.632732:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077766.632736:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077766.632744:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616792960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077766.632750:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077766.632753:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077766.632755:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077766.632758:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4752 00000100:00080000:0.0:1769077766.632760:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077766.632762:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077766.632765:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077766.632782:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616793088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077766.632787:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077766.632789:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077766.632791:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077766.632794:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4752 00000100:00080000:0.0:1769077766.632797:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077766.632799:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077766.632801:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077766.632808:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077766.632812:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616793216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077766.632819:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077766.632822:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077766.632824:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077766.632827:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4752 00000100:00080000:0.0:1769077766.632829:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077766.632832:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077766.632846:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077766.632855:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616793344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077766.632860:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077766/real 1769077766] req@ffff995b84643800 x1855007616792704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077766.632864:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077766.632871:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077766.632875:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077766.632881:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4752 00000100:00080000:0.0:1769077766.632884:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077766.632888:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616792704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077766.632890:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077766.632897:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077766.632903:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077766.632908:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616793472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077766.632911:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077766.632919:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616792704:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077766.632945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077766.632964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077766.632976:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077766/real 1769077766] req@ffff995b84641180 x1855007616792832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077766.632987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616792832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077766.632994:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077766.632998:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077766.633003:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077766.633009:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077766.633022:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077766.633028:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077766/real 1769077766] req@ffff995b84641500 x1855007616792960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077766.633037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616792960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077766.633044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077766.633046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077766.633047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616792960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077766.633051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077766.633062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077766.633067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077766/real 1769077766] req@ffff995b84640000 x1855007616793088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077766.633075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616793088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077766.633081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077766.633082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077766.633084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077766.633087:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793216:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077766.633097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077766.633103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077766/real 1769077766] req@ffff995b84640380 x1855007616793216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077766.633110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616793216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077766.633116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077766.633118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077766.633120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793216:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077766.633123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616793344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077766.633132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077766.633138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077766/real 1769077766] req@ffff995b84641f80 x1855007616793344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077766.633145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616793344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077766.633150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077766.633152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077766.633154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616793344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077766.633157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616793472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077766.633166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077766.633172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077766/real 1769077766] req@ffff995b84640a80 x1855007616793472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077766.633184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616793472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077766.633189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077766.633191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077766.633193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616793472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077767.336542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077767.336607:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077767.656570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077767.656581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077767.656602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077767.656613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077768.360593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077768.360661:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077768.680602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077768.680610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077768.680623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077768.680656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077769.384562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077769.384646:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077769.704586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077769.704594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077769.704619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077769.704625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077770.408557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077770.408628:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077770.728521:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077770.728550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077770.728554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077770.728555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077771.432545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077771.432616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077771.752594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077771.752594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077771.752600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077771.752714:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077771.752722:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077771.752727:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077771.752734:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077771.752739:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4757 00000100:00080000:2.0:1769077771.752754:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077771.752763:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077771.752812:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616793600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077771.752848:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077771.752853:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077771.752855:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077771.752861:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4757 00000100:00080000:2.0:1769077771.752865:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077771.752871:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077771.752881:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616793600:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077771.752883:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077771.752903:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616793728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077771.752915:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077771.752919:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077771.752922:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077771.752926:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4757 00000100:00080000:2.0:1769077771.752931:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077771.752936:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077771.752942:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077771.752957:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616793856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077771.752958:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077771.752972:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077771.752976:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077771.752979:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077771.752984:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4757 00000100:00080000:2.0:1769077771.752989:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077771.752995:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077771.753000:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077771/real 1769077771] req@ffff995b85acb800 x1855007616793600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077771.753003:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077771.753022:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616793984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077771.753024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616793600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077771.753054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077771.753057:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077771.753061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077771.753068:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1769077771.753069:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616793600:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077771.753074:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077771.753081:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4757 00000100:00080000:2.0:1769077771.753087:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077771.753092:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077771.753098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793728:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077771.753098:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077771.753119:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616794112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077771.753133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077771.753133:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077771.753140:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077771.753143:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077771.753147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077771/real 1769077771] req@ffff995b85ac8a80 x1855007616793728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077771.753159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616793728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077771.753162:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4757 00000100:00080000:0.0:1769077771.753173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077771.753173:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077771.753180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077771.753185:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077771.753186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793728:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077771.753196:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077771.753197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077771.753219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077771.753220:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616794240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077771.753228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077771/real 1769077771] req@ffff995b85acbb80 x1855007616793856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077771.753254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616793856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077771.753261:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077771.753265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077771.753267:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077771.753270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077771.753273:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077771.753274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793856:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077771.753286:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4757 00000100:00100000:0.0:1769077771.753287:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793984:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769077771.753295:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077771.753303:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077771.753308:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077771.753313:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077771.753333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077771/real 1769077771] req@ffff995b85ac9880 x1855007616793984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077771.753347:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616794368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077771.753352:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616793984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077771.753387:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077771.753391:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077771.753394:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616793984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077771.753411:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616794112:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077771.753428:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077771.753436:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077771/real 1769077771] req@ffff995b85ac9c00 x1855007616794112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077771.753444:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616794112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077771.753451:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077771.753473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077771.753476:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616794112:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077771.753480:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616794240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077771.753493:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077771.753501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077771/real 1769077771] req@ffff995b85acb100 x1855007616794240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077771.753514:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616794240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077771.753520:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077771.753522:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077771.753524:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616794240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077771.753528:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616794368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077771.753537:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077771.753543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077771/real 1769077771] req@ffff995b85ac8380 x1855007616794368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077771.753551:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616794368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077771.753556:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077771.753558:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077771.753560:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616794368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077772.456572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077772.456633:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077772.776572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077772.776589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077772.776611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077772.776619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077773.480593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077773.480613:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077773.800574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077773.800581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077773.800727:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077773.800731:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077774.504583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077774.504645:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077774.824538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077774.824586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077774.824625:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077774.824648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077775.528587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077775.528656:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077775.848586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077775.848681:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077775.848694:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077775.848740:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077776.552595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077776.552656:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077776.872666:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077776.872667:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077776.872671:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077776.872672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077776.872687:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077776.872693:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077776.872704:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4762 00000100:00080000:1.0:1769077776.872725:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077776.872738:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077776.872739:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077776.872795:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616794496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077776.872822:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077776.872826:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077776.872828:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077776.872833:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4762 00000100:00080000:1.0:1769077776.872838:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077776.872842:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077776.872851:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077776.872862:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616794624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077776.872871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616794496:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077776.872889:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077776.872900:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077776.872901:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077776.872906:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4762 00000100:00080000:1.0:1769077776.872908:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077776.872910:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077776.872914:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077776.872925:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616794752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077776.872932:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077776.872935:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077776.872936:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077776.872938:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4762 00000100:00080000:1.0:1769077776.872940:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077776.872943:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077776.872946:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077776.872955:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616794880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077776.872961:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077776.872964:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077776.872965:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077776.872968:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4762 00000100:00080000:1.0:1769077776.872972:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077776.872976:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077776.872979:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077776.872987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077776.872989:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616795008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077776.872997:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077776.873000:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077776.873002:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077776.873007:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4762 00000100:00080000:1.0:1769077776.873016:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077776.873020:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077776.873023:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077776.873044:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616795136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077776.873048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077776/real 1769077776] req@ffff995b868c7100 x1855007616794496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077776.873054:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077776.873063:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077776.873065:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077776.873075:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4762 00000100:00080000:1.0:1769077776.873085:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077776.873099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616794496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077776.873101:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077776.873114:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077776.873122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077776.873131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077776.873133:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616795264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077776.873143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616794496:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077776.873185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616794624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077776.873210:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077776.873224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077776/real 1769077776] req@ffff995b868c6a00 x1855007616794624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077776.873254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616794624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077776.873264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077776.873269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077776.873273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616794624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077776.873280:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616794752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077776.873302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077776.873314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077776/real 1769077776] req@ffff995b868c5c00 x1855007616794752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077776.873328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616794752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077776.873340:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077776.873344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077776.873348:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616794752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077776.873355:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616794880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077776.873373:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077776.873381:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077776/real 1769077776] req@ffff995b868c7480 x1855007616794880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077776.873389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616794880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077776.873400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077776.873403:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077776.873405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616794880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077776.873410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795008:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077776.873420:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077776.873426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077776/real 1769077776] req@ffff995b868c7800 x1855007616795008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077776.873434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616795008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077776.873441:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077776.873443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077776.873444:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077776.873448:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616795136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077776.873485:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077776.873493:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077776/real 1769077776] req@ffff995b868c5500 x1855007616795136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077776.873501:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616795136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077776.873507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077776.873509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077776.873512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616795136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077776.873516:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616795264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077776.873527:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077776.873533:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077776/real 1769077776] req@ffff995b868c4700 x1855007616795264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077776.873541:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616795264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077776.873546:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077776.873548:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077776.873550:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616795264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077777.576570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077777.576641:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077777.896541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077777.896570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077777.896575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077777.896588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077778.600583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077778.600647:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077778.920568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077778.920579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077778.920715:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077778.920803:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077779.624580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077779.624752:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077779.944553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077779.944553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077779.944554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077779.944563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077780.648566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077780.648621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077780.968570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077780.968570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077780.968570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077780.968570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077781.672592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077781.672656:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077781.992579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077781.992607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077781.992659:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077781.992664:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077781.992668:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077781.992677:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4767 00000100:00100000:0.0:1769077781.992690:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077781.992690:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077781.992699:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077781.992734:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077781.992750:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616795392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077781.992783:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077781.992789:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077781.992791:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077781.992795:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4767 00000100:00080000:3.0:1769077781.992797:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077781.992802:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077781.992806:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616795392:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769077781.992812:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077781.992828:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616795520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077781.992836:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077781.992838:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077781.992840:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077781.992842:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4767 00000100:00080000:3.0:1769077781.992845:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077781.992848:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077781.992852:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077781.992874:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616795648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077781.992883:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077781.992887:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077781.992889:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077781.992894:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077781.992897:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4767 00000100:00080000:3.0:1769077781.992905:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077781.992909:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077781.992916:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077781.992937:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616795776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077781.992944:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077781.992946:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077781.992948:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077781.992951:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4767 00000100:00080000:3.0:1769077781.992955:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077781.992959:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077781.992963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077781/real 1769077781] req@ffff995b84559880 x1855007616795392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077781.992965:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077781.992987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616795392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077781.992996:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077781.993001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077781.993013:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616795392:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077781.993041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077781.993077:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616795904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077781.993087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077781.993105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077781/real 1769077781] req@ffff995b84559c00 x1855007616795520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077781.993115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616795520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077781.993131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077781.993133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077781.993137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077781.993141:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077781.993174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077781.993183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077781/real 1769077781] req@ffff995b8455a300 x1855007616795648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077781.993194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616795648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077781.993205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077781.993209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077781.993213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077781.993221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795776:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077781.993245:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077781.993255:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077781.993259:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077781.993261:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077781.993264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077781/real 1769077781] req@ffff995b8455ad80 x1855007616795776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077781.993282:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4767 00000100:00080000:3.0:1769077781.993290:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077781.993296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616795776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077781.993305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077781.993311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077781.993315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077781.993338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795904:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077781.993352:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769077781.993358:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769077781.993363:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077781.993375:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077781/real 1769077781] req@ffff995b8455b480 x1855007616795904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077781.993382:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616796032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077781.993405:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077781.993409:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077781.993412:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077781.993414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616795904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077781.993434:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4767 00000100:00080000:1.0:1769077781.993438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077781.993441:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077781.993446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077781.993447:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077781.993474:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077781.993494:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616795904:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077781.993496:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616796160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077781.993518:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616796032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077781.993552:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077781.993568:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077781/real 1769077781] req@ffff995b84558700 x1855007616796032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077781.993582:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616796032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077781.993591:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077781.993594:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077781.993597:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616796032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077781.993601:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616796160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077781.993617:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077781.993627:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077781/real 1769077781] req@ffff995b8455ad80 x1855007616796160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077781.993639:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616796160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077781.993648:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077781.993652:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077781.993656:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616796160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077782.696542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077782.696661:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077783.016632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077783.016654:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077783.016680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077783.016704:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769077783.720658:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077783.720749:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077784.040538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077784.040576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077784.040618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077784.040651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077784.744559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077784.744651:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077785.064604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077785.064625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077785.064625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077785.064665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077785.768573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077785.768633:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077786.088568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077786.088580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077786.088584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077786.088618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077786.792548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077786.792669:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077787.112537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077787.112550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077787.112561:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077787.112568:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077787.112574:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077787.112583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077787.112585:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4772 00000100:00100000:3.0:1769077787.112588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077787.112599:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077787.112609:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077787.112652:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616796288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077787.112680:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077787.112683:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077787.112685:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077787.112688:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4772 00000100:00080000:2.0:1769077787.112692:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077787.112695:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077787.112704:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077787.112711:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616796288:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077787.112717:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616796416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077787.112725:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077787.112729:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077787.112730:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077787.112733:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4772 00000100:00080000:2.0:1769077787.112736:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077787.112738:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077787.112742:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077787.112757:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616796544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077787.112764:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077787.112766:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077787.112767:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077787.112769:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4772 00000100:00080000:2.0:1769077787.112771:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077787.112773:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769077787.112778:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077787.112778:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077787.112788:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616796672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077787.112795:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077787.112798:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077787.112799:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077787.112802:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4772 00000100:00080000:2.0:1769077787.112804:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077787.112806:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077787.112810:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077787.112822:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616796800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077787.112823:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077787/real 1769077787] req@ffff995b85aca300 x1855007616796288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077787.112837:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077787.112844:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077787.112846:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077787.112851:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4772 00000100:00100000:1.0:1769077787.112855:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616796288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077787.112856:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077787.112865:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769077787.112869:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769077787.112872:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769077787.112875:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077787.112881:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616796288:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077787.112890:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616796928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077787.112903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616796416:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077787.112912:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077787.112917:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077787.112919:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077787.112926:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077787.112927:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4772 00000100:00080000:2.0:1769077787.112935:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077787.112942:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077787/real 1769077787] req@ffff995b85acaa00 x1855007616796416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077787.112943:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077787.112954:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077787.112958:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616796416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077787.112966:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077787.112972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077787.112973:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616797056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077787.112977:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616796416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077787.112987:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616796544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077787.113003:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077787.113011:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077787/real 1769077787] req@ffff995b85ac8380 x1855007616796544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077787.113019:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616796544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077787.113026:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077787.113028:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077787.113030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616796544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077787.113034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616796672:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077787.113044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077787.113070:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077787/real 1769077787] req@ffff995b85acb100 x1855007616796672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077787.113081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616796672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077787.113088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077787.113091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077787.113093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616796672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077787.113097:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616796800:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077787.113111:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077787.113118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077787/real 1769077787] req@ffff995b85ac9c00 x1855007616796800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077787.113125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616796800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077787.113131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077787.113133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077787.113135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616796800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077787.113146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616796928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077787.113156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077787.113162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077787/real 1769077787] req@ffff995b85ac9880 x1855007616796928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077787.113169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616796928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077787.113174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077787.113176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077787.113179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616796928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077787.113182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616797056:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077787.113192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077787.113198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077787/real 1769077787] req@ffff995b85acbb80 x1855007616797056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077787.113212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616797056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077787.113218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077787.113220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077787.113222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616797056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077787.816575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077787.816641:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077788.136533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077788.136549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077788.136581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077788.136600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077788.840543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077788.840661:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077789.160565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077789.160577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077789.160625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077789.160637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077789.864620:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077789.864711:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077790.184546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077790.184563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077790.184595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077790.184621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077790.888525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077790.888625:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077791.208606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077791.208618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077791.208620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077791.208663:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077791.912599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077791.912683:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077792.232533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077792.232550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077792.232604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077792.232616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077792.232619:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077792.232626:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077792.232632:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077792.232642:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4777 00000100:00080000:1.0:1769077792.232652:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077792.232659:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077792.232703:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616797184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077792.232733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077792.232737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077792.232738:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077792.232742:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4777 00000100:00080000:1.0:1769077792.232746:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077792.232750:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077792.232760:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077792.232765:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616797184:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077792.232774:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616797312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077792.232783:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077792.232786:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077792.232787:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077792.232790:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4777 00000100:00080000:1.0:1769077792.232792:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077792.232795:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077792.232798:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077792.232806:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616797440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077792.232812:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077792.232814:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077792.232816:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077792.232819:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4777 00000100:00080000:1.0:1769077792.232823:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077792.232828:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077792.232829:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077792.232837:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077792.232852:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616797568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077792.232863:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077792.232864:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077792/real 1769077792] req@ffff995b868c6300 x1855007616797184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077792.232870:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077792.232876:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077792.232883:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4777 00000100:00100000:3.0:1769077792.232886:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616797184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077792.232888:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077792.232896:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077792.232903:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769077792.232907:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077792.232912:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077792.232917:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616797696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077792.232918:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616797184:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077792.232931:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077792.232934:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077792.232936:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077792.232940:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4777 00000100:00080000:1.0:1769077792.232945:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077792.232945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616797312:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077792.232953:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077792.232959:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077792.232969:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077792.232987:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616797824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077792.232996:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077792/real 1769077792] req@ffff995b868c6d80 x1855007616797312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077792.233016:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077792.233020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616797312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077792.233022:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077792.233027:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077792.233033:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077792.233036:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4777 00000100:00080000:3.0:1769077792.233039:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077792.233044:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616797312:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077792.233046:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077792.233051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616797440:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077792.233056:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077792.233065:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077792.233071:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077792.233079:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616797952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077792.233087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077792/real 1769077792] req@ffff995b868c4a80 x1855007616797440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077792.233098:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616797440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077792.233106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077792.233109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077792.233111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616797440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077792.233117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616797568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077792.233137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077792.233149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077792/real 1769077792] req@ffff995b868c4700 x1855007616797568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077792.233160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616797568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077792.233168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077792.233171:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077792.233174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616797568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077792.233179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616797696:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077792.233196:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077792.233207:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077792/real 1769077792] req@ffff995b868c5500 x1855007616797696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077792.233218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616797696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077792.233228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077792.233231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077792.233234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616797696:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077792.233249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616797824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077792.233268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077792.233278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077792/real 1769077792] req@ffff995b868c7800 x1855007616797824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077792.233289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616797824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077792.233297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077792.233299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077792.233302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616797824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077792.233308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616797952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077792.233324:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077792.233335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077792/real 1769077792] req@ffff995b868c7480 x1855007616797952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077792.233346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616797952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077792.233363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077792.233367:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077792.233371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616797952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077792.936512:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077792.936658:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077793.256600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077793.256600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077793.256622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077793.256655:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077793.960586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077793.960672:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077794.280577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077794.280577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077794.280614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077794.280725:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077794.984581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077794.984643:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077795.304592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077795.304592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077795.304619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077795.304624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077796.008584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077796.008675:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077796.328619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077796.328620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077796.328628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077796.328711:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077797.032609:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077797.032678:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077797.352552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077797.352552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077797.352554:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077797.352565:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077797.352570:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077797.352595:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4783 00000100:00080000:1.0:1769077797.352610:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077797.352622:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077797.352650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077797.352658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077797.352674:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616798080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077797.352700:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077797.352704:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077797.352707:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077797.352712:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4783 00000100:00080000:1.0:1769077797.352716:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077797.352720:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077797.352741:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077797.352747:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616798080:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077797.352755:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616798208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077797.352765:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077797.352767:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077797.352769:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077797.352772:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4783 00000100:00080000:1.0:1769077797.352775:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077797.352777:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077797.352780:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077797.352789:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616798336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077797.352795:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077797.352797:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077797.352799:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077797.352802:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4783 00000100:00080000:1.0:1769077797.352805:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077797.352807:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077797.352810:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077797.352821:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616798464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077797.352846:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077797.352850:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077797.352852:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077797.352855:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4783 00000100:00080000:1.0:1769077797.352859:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077797.352863:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077797.352869:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077797.352902:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616798592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077797.352922:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077797.352942:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077797.352944:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077797.352952:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4783 00000100:00080000:1.0:1769077797.352956:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077797.352961:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616798208:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077797.352966:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077797.352975:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077797.352994:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616798720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077797.353003:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077797.353008:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077797.353010:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077797.353016:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616798336:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077797.353020:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4783 00000100:00080000:1.0:1769077797.353030:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077797.353034:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077797.353039:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077797.353050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616798464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077797.353056:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616798848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077797.353090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616798592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077797.353119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616798720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077797.353163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616798848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077798.056579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077798.056646:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077798.376595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077798.376597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077798.376611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077798.376668:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077799.080601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077799.080703:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077799.400592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077799.400620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077799.400638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077799.400652:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077800.104588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077800.104670:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077800.424627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077800.424627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077800.424628:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077800.424813:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000400:00000100:3.0:1769077800.424858:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:3.0:1769077800.424919:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769077800.424938:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077800.424970:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077800.424977:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077800.424984:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077800.424991:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077800.424999:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077800.425005:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:1.0:1769077800.425016:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077797/real 1769077800] req@ffff995b868c6a00 x1855007616798208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077800.425048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616798208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077800.425061:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077800.425067:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077800.425072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616798208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077800.425099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077797/real 1769077800] req@ffff995b868c7100 x1855007616798336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077800.425114:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616798336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077800.425124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077800.425128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077800.425132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616798336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077800.425141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077797/real 1769077800] req@ffff995b868c7480 x1855007616798464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077800.425157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616798464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077800.425165:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077800.425168:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077800.425171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616798464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077800.425178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077797/real 1769077800] req@ffff995b868c7800 x1855007616798592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077800.425191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616798592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077800.425199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077800.425202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077800.425205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616798592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077800.425213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077797/real 1769077800] req@ffff995b868c5c00 x1855007616798080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077800.425226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616798080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077800.425233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077800.425237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077800.425240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616798080:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077800.425271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077797/real 1769077800] req@ffff995b868c5500 x1855007616798720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077800.425284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616798720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077800.425293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077800.425297:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077800.425300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616798720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077800.425308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077797/real 1769077800] req@ffff995b868c4700 x1855007616798848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077800.425320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616798848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077800.425328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077800.425331:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077800.425335:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616798848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077801.128568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077801.128649:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077801.448637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077801.448639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077801.448666:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077801.448703:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077802.152600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077802.152758:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077802.472567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077802.472636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077802.472649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077802.472667:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077802.472679:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769077802.472685:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077802.472688:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077802.472710:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4788 00000100:00080000:3.0:1769077802.472730:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077802.472744:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077802.472836:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616798976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077802.472884:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077802.472892:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077802.472895:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077802.472903:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4788 00000100:00080000:3.0:1769077802.472911:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077802.472920:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077802.472937:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077802.472964:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616799104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077802.472989:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077802.472996:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077802.473001:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077802.473009:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616798976:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077802.473018:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4788 00000100:00080000:3.0:1769077802.473033:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077802.473041:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077802.473052:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077802.473080:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616799232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077802.473113:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077802.473122:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077802.473128:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077802.473136:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4788 00000100:00080000:3.0:1769077802.473145:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077802.473155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077802.473164:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077802.473181:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077802.473208:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616799360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077802.473229:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077802.473241:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077802.473242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077802/real 1769077802] req@ffff995b8455a300 x1855007616798976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077802.473247:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077802.473267:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4788 00000100:00080000:3.0:1769077802.473283:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077802.473296:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077802.473308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616798976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077802.473309:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077802.473344:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616799488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077802.473355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077802.473375:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077802.473386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616798976:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077802.473402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616799104:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077802.473408:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077802.473418:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077802.473422:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077802.473432:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4788 00000100:00080000:3.0:1769077802.473444:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077802.473505:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077802.473540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077802/real 1769077802] req@ffff995b84559c00 x1855007616799104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077802.473545:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077802.473561:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077802.473604:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616799616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077802.473607:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616799104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077802.473672:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077802.473673:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077802.473690:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077802.473692:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077802.473704:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077802.473709:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616799104:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077802.473720:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4788 00000100:00080000:3.0:1769077802.473733:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077802.473744:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077802.473757:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077802.473772:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616799232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077802.473787:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616799744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077802.473857:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077802.473886:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077802/real 1769077802] req@ffff995b84559880 x1855007616799232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077802.473912:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616799232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077802.473931:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077802.473938:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077802.473944:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616799232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077802.473959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616799360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077802.474000:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077802.474025:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077802/real 1769077802] req@ffff995b84558000 x1855007616799360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077802.474051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616799360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077802.474067:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077802.474074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077802.474080:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616799360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077802.474092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616799488:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077802.474128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077802.474148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077802/real 1769077802] req@ffff995b84559500 x1855007616799488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077802.474170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616799488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077802.474188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077802.474196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077802.474204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616799488:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077802.474217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616799616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077802.474255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077802.474277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077802/real 1769077802] req@ffff995b8455bb80 x1855007616799616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077802.474302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616799616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077802.474319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077802.474326:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077802.474334:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616799616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077802.475148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616799744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077802.475237:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077802.475274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077802/real 1769077802] req@ffff995b84558a80 x1855007616799744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077802.475297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616799744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077802.475312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077802.475317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077802.475322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616799744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077803.176560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077803.176591:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077803.496619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077803.496659:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077803.496678:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077803.496719:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077804.200543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077804.200635:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077804.520608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077804.520623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077804.520623:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077804.520623:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077805.224588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077805.224657:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077805.544557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077805.544557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077805.544570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077805.544612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077806.248535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077806.248629:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077806.568589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077806.568600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077806.568611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077806.568630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077807.272578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077807.272653:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077807.592582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077807.592582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077807.592591:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077807.592609:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769077807.592609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077807.592616:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077807.592626:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4793 00000100:00080000:0.0:1769077807.592637:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077807.592643:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077807.592645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077807.592681:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616799872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077807.592703:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077807.592706:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077807.592708:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077807.592711:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4793 00000100:00080000:0.0:1769077807.592726:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077807.592730:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077807.592739:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077807.592750:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616800000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077807.592755:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616799872:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077807.592762:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077807.592780:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077807.592782:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077807.592787:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4793 00000100:00080000:0.0:1769077807.592791:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077807.592796:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077807.592801:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077807.592824:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616800128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077807.592835:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077807.592838:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077807.592839:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077807.592843:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4793 00000100:00080000:0.0:1769077807.592849:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077807.592849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077807.592867:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077807.592872:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077807.592884:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616800256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077807.592891:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077807.592894:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077807.592895:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077807.592898:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4793 00000100:00080000:0.0:1769077807.592902:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077807.592904:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077807.592907:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077807.592912:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077807/real 1769077807] req@ffff995b84641c00 x1855007616799872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077807.592918:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616800384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077807.592954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077807.592958:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077807.592960:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077807.592965:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4793 00000100:00100000:1.0:1769077807.592969:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616799872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077807.592971:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077807.592981:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769077807.592984:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769077807.592988:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769077807.592990:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077807.592997:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616799872:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077807.593005:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616800512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077807.593017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077807.593021:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769077807.593022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800000:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077807.593024:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077807.593032:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4793 00000100:00080000:0.0:1769077807.593037:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077807.593052:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077807.593058:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077807.593059:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077807.593073:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077807/real 1769077807] req@ffff995b84640e00 x1855007616800000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077807.593078:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616800640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077807.593092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616800000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077807.593114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077807.593119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077807.593123:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077807.593128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077807.593141:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077807.593148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077807/real 1769077807] req@ffff995b84642680 x1855007616800128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077807.593162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616800128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077807.593170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077807.593173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077807.593175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077807.593179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077807.593191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077807.593201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077807/real 1769077807] req@ffff995b84640a80 x1855007616800256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077807.593214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616800256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077807.593223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077807.593227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077807.593230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077807.593237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800384:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077807.593256:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077807.593265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077807/real 1769077807] req@ffff995b84641f80 x1855007616800384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077807.593272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616800384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077807.593283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077807.593286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077807.593288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800384:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077807.593292:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616800512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077807.593303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077807.593309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077807/real 1769077807] req@ffff995b84640380 x1855007616800512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077807.593316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616800512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077807.593322:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077807.593324:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077807.593326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616800512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077807.593340:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616800640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077807.593350:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077807.593356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077807/real 1769077807] req@ffff995b84640000 x1855007616800640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077807.593363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616800640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077807.593368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077807.593370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077807.593373:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616800640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077808.296537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077808.296606:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077808.616574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077808.616574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077808.616574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077808.616585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077809.320558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077809.320635:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077809.640536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077809.640562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077809.640874:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077809.640905:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077810.344532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077810.344583:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077810.664526:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077810.664553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077810.664571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077810.664591:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077811.368550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077811.368605:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077811.688530:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077811.688551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077811.688632:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077811.688660:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077812.392540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077812.392692:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077812.712505:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077812.712521:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077812.712534:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077812.712570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077812.712586:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077812.712595:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077812.712599:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077812.712607:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4798 00000100:00080000:3.0:1769077812.712619:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077812.712638:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077812.712679:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616800768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077812.712707:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077812.712709:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077812.712711:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077812.712714:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4798 00000100:00080000:3.0:1769077812.712718:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077812.712722:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077812.712730:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077812.712741:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616800768:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077812.712744:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616800896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077812.712754:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077812.712756:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077812.712758:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077812.712761:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4798 00000100:00080000:3.0:1769077812.712763:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077812.712766:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077812.712769:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077812.712778:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616801024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077812.712784:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077812.712787:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077812.712789:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077812.712794:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4798 00000100:00080000:3.0:1769077812.712798:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077812.712802:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077812.712808:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769077812.712809:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077812.712827:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616801152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077812.712838:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077812.712840:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077812.712842:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077812.712845:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4798 00000100:00080000:3.0:1769077812.712850:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077812.712854:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077812/real 1769077812] req@ffff995b8455a680 x1855007616800768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077812.712856:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077812.712863:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077812.712876:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616801280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077812.712880:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616800768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077812.712898:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077812.712901:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077812.712902:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077812.712906:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077812.712909:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077812.712913:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4798 00000100:00080000:3.0:1769077812.712917:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077812.712918:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616800768:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077812.712923:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077812.712929:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077812.712937:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616801408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077812.712951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077812.712955:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769077812.712958:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800896:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077812.712958:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077812.712968:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4798 00000100:00080000:3.0:1769077812.712974:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077812.712979:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077812.712986:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077812.712990:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077812.713005:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616801536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077812.713006:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077812/real 1769077812] req@ffff995b8455b100 x1855007616800896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077812.713041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616800896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077812.713047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077812.713050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077812.713052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616800896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077812.713057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077812.713069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077812.713076:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077812/real 1769077812] req@ffff995b8455aa00 x1855007616801024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077812.713084:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616801024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077812.713091:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077812.713093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077812.713095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077812.713099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801152:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077812.713110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077812.713115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077812/real 1769077812] req@ffff995b84558a80 x1855007616801152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077812.713123:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616801152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077812.713134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077812.713136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077812.713138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077812.713142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801280:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077812.713151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077812.713156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077812/real 1769077812] req@ffff995b8455bb80 x1855007616801280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077812.713163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616801280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077812.713169:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077812.713171:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077812.713173:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077812.713184:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616801408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077812.713199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077812.713210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077812/real 1769077812] req@ffff995b84559500 x1855007616801408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077812.713223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616801408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077812.713232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077812.713236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077812.713240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616801408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077812.713246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616801536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077812.713263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077812.713271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077812/real 1769077812] req@ffff995b84558000 x1855007616801536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077812.713279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616801536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077812.713285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077812.713287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077812.713289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616801536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077813.416563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077813.416643:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077813.736537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077813.736548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077813.736549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077813.736582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077814.440587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077814.440667:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077814.760527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077814.760539:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077814.760553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077814.760567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077815.464608:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077815.464703:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077815.784595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077815.784595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077815.784596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077815.784615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077816.488610:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077816.488683:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077816.808588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077816.808602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077816.808631:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077816.808699:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077817.512695:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077817.512808:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077817.832573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077817.832605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077817.832631:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077817.832643:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077817.832650:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077817.832659:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4803 00000100:00100000:2.0:1769077817.832668:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077817.832671:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077817.832680:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077817.832703:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077817.832718:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616801664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077817.832746:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077817.832749:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077817.832751:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077817.832756:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4803 00000100:00080000:3.0:1769077817.832759:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077817.832763:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077817.832774:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077817.832782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616801664:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077817.832789:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616801792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077817.832806:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077817.832809:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077817.832811:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077817.832819:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4803 00000100:00080000:3.0:1769077817.832826:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077817.832829:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077817.832834:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077817.832850:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616801920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077817.832857:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077817.832862:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077817.832863:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077817.832867:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077817.832871:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4803 00000100:00080000:3.0:1769077817.832874:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077817.832877:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077817.832881:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077817.832903:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077817/real 1769077817] req@ffff995b84559880 x1855007616801664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077817.832918:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616801664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077817.832927:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077817.832931:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077817.832934:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616801664:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077817.832952:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616802048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077817.832957:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801792:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077817.832970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077817.832973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077817.832975:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077817.832979:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4803 00000800:00000100:2.0:1769077817.832982:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077817.832985:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077817.832991:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077817.832997:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077817/real 1769077817] req@ffff995b84559c00 x1855007616801792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077817.832997:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077817.833007:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616801792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077817.833013:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077817.833016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077817.833018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077817.833022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077817.833033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077817.833039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077817/real 1769077817] req@ffff995b8455a300 x1855007616801920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077817.833047:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616801920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077817.833060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077817.833062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077817.833064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616801920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077817.833074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077817.833086:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077817.833093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077817/real 1769077817] req@ffff995b8455ad80 x1855007616802048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077817.833112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616802048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077817.833119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077817.833122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077817.833124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077817.833228:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616802176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077817.833255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077817.833260:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077817.833262:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077817.833270:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4803 00000100:00080000:3.0:1769077817.833277:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077817.833284:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077817.833295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802176:192.168.204.132@tcp:8:kworker.0 00000008:00100000:3.0:1769077817.833295:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077817.833317:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616802304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077817.833329:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077817.833333:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077817.833336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077817.833339:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077817.833345:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4803 00000100:00080000:3.0:1769077817.833349:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077817.833350:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077817/real 1769077817] req@ffff995b84558700 x1855007616802176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077817.833354:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077817.833363:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077817.833378:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616802432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077817.833390:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616802176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077817.833403:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077817.833406:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077817.833409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802176:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077817.833419:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616802304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077817.833433:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077817.833440:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077817/real 1769077817] req@ffff995b8455b480 x1855007616802304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077817.833447:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616802304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077817.833474:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077817.833477:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077817.833483:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616802304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077817.833492:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616802432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077817.833516:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077817.833525:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077817/real 1769077817] req@ffff995b8455ad80 x1855007616802432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077817.833532:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616802432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077817.833538:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077817.833555:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077817.833557:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616802432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077818.536580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077818.536657:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077818.856556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077818.856572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077818.856599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077818.856604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077819.560578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077819.560650:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077819.880602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077819.880602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077819.880622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077819.880647:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077820.584576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077820.584632:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077820.904570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077820.904570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077820.904571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077820.904612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077821.608553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077821.608617:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077821.928592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077821.928616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077821.928750:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077821.928781:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077822.632581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077822.632649:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077822.952562:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077822.952567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077822.952570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077822.952572:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077822.952579:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077822.952588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077822.952589:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4808 00000100:00080000:1.0:1769077822.952602:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077822.952609:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077822.952618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077822.952648:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616802560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077822.952670:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077822.952673:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077822.952674:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077822.952677:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4808 00000100:00080000:1.0:1769077822.952680:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077822.952682:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077822.952691:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077822.952700:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616802688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077822.952708:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077822.952711:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077822.952714:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077822.952715:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616802560:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077822.952718:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4808 00000100:00080000:1.0:1769077822.952723:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077822.952728:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077822.952732:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077822.952740:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616802816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077822.952747:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077822.952749:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077822.952751:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077822.952753:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4808 00000100:00080000:1.0:1769077822.952755:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077822.952757:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077822.952761:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077822.952767:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616802944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077822.952773:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077822.952776:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077822.952778:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077822.952782:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4808 00000100:00080000:1.0:1769077822.952787:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077822.952791:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077822.952796:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077822.952798:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077822.952813:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616803072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077822.952835:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077822.952839:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077822.952843:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077822.952844:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077822/real 1769077822] req@ffff995b868c4700 x1855007616802560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077822.952852:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4808 00000100:00080000:1.0:1769077822.952860:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077822.952868:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077822.952873:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077822.952875:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616802560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077822.952888:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616803200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077822.952890:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077822.952900:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077822.952905:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077822.952910:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616802560:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077822.952911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077822.952916:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077822.952924:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4808 00000100:00080000:1.0:1769077822.952927:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077822.952939:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077822.952945:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077822.952945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077822.952962:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616803328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077822.952980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077822.952998:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077822/real 1769077822] req@ffff995b868c5500 x1855007616802688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077822.953013:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616802688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077822.953021:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077822.953025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077822.953029:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077822.953039:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077822.953059:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077822.953071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077822/real 1769077822] req@ffff995b868c5c00 x1855007616802816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077822.953084:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616802816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077822.953095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077822.953098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077822.953102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077822.953108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802944:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077822.953126:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077822.953136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077822/real 1769077822] req@ffff995b868c7800 x1855007616802944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077822.953147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616802944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077822.953156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077822.953160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077822.953164:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616802944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077822.953170:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803072:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077822.953189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077822.953200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077822/real 1769077822] req@ffff995b868c7480 x1855007616803072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077822.953212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616803072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077822.953221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077822.953225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077822.953228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077822.953233:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616803200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077822.953253:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077822.953265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077822/real 1769077822] req@ffff995b868c7100 x1855007616803200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077822.953278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616803200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077822.953286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077822.953289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077822.953292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616803200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077822.953310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616803328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077822.953329:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077822.953341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077822/real 1769077822] req@ffff995b868c6a00 x1855007616803328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077822.953353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616803328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077822.953368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077822.953371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077822.953375:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616803328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077823.656534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077823.656592:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077823.976581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077823.976582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077823.976590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077823.976613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077824.680557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077824.680620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077825.000512:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077825.000523:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077825.000566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077825.000579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077825.704527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077825.704600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077826.024561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077826.024563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077826.024688:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077826.024695:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077826.728567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077826.728633:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077827.048541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077827.048554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077827.048571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077827.048573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077827.752510:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077827.752573:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077828.072564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077828.072565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077828.072565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077828.072574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077828.072609:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077828.072616:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077828.072621:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077828.072632:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4813 00000100:00080000:2.0:1769077828.072644:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077828.072651:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077828.072692:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616803456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077828.072717:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077828.072720:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077828.072721:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077828.072725:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4813 00000100:00080000:2.0:1769077828.072729:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077828.072732:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077828.072740:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077828.072750:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616803584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077828.072758:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077828.072762:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769077828.072765:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616803456:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077828.072765:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077828.072774:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4813 00000100:00080000:2.0:1769077828.072777:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077828.072779:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077828.072783:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077828.072795:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616803712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077828.072802:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077828.072805:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077828.072806:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077828.072809:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4813 00000100:00080000:2.0:1769077828.072812:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077828.072814:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077828.072817:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077828.072826:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616803840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077828.072833:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077828.072853:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077828.072856:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077828.072857:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077828.072861:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4813 00000100:00080000:2.0:1769077828.072864:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077828.072867:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077828.072871:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077828.072872:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077828/real 1769077828] req@ffff995b85ac8a80 x1855007616803456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077828.072884:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616803968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077828.072892:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077828.072895:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616803456/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077828.072896:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077828.072899:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077828.072907:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4813 00000100:00080000:2.0:1769077828.072913:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077828.072915:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077828.072916:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077828.072924:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769077828.072925:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077828.072931:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616803456:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077828.072935:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616804096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077828.072943:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077828.072945:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077828.072947:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077828.072949:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4813 00000100:00080000:2.0:1769077828.072953:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077828.072957:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803584:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077828.072957:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077828.072965:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077828.072976:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616804224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077828.072985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077828.073000:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077828/real 1769077828] req@ffff995b85acb800 x1855007616803584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077828.073010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616803584/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077828.073016:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077828.073018:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077828.073021:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077828.073026:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077828.073041:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077828.073049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077828/real 1769077828] req@ffff995b85acbb80 x1855007616803712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077828.073057:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616803712/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077828.073068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077828.073070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077828.073073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077828.073076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803840:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077828.073087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077828.073093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077828/real 1769077828] req@ffff995b85ac9880 x1855007616803840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077828.073100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616803840/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077828.073105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077828.073107:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077828.073109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803840:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077828.073112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803968:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077828.073122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077828.073127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077828/real 1769077828] req@ffff995b85ac9c00 x1855007616803968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077828.073134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616803968/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077828.073140:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077828.073142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077828.073144:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616803968:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077828.073148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616804096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077828.073157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077828.073163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077828/real 1769077828] req@ffff995b85acb100 x1855007616804096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077828.073170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616804096/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077828.073175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077828.073177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077828.073179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616804096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077828.073192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616804224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077828.073202:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077828.073208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077828/real 1769077828] req@ffff995b85ac8380 x1855007616804224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077828.073214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616804224/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077828.073220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077828.073221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077828.073223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616804224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077828.776531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077828.776603:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077829.096588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077829.096605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077829.096611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077829.096614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077829.800605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077829.800684:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077830.120581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077830.120594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077830.120641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077830.120658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077830.824574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077830.824683:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077831.144565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077831.144567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077831.144602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077831.144634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077831.848552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077831.848644:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077832.168522:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077832.168557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077832.168570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077832.168597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077832.872603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077832.872634:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077833.192560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077833.192561:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077833.192572:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769077833.192574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077833.192578:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077833.192589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077833.192591:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4818 00000100:00080000:0.0:1769077833.192606:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077833.192606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077833.192617:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077833.192661:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616804352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077833.192687:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077833.192690:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077833.192692:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077833.192696:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4818 00000100:00080000:0.0:1769077833.192699:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077833.192703:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077833.192711:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077833.192718:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616804352:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077833.192723:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616804480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077833.192732:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077833.192734:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077833.192735:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077833.192738:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4818 00000100:00080000:0.0:1769077833.192740:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077833.192743:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077833.192746:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077833.192755:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616804608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077833.192761:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077833.192763:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077833.192764:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077833.192766:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4818 00000100:00080000:0.0:1769077833.192775:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077833.192777:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077833.192781:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077833.192782:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077833.192790:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616804736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077833.192798:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077833.192800:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077833.192801:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077833.192804:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4818 00000100:00080000:0.0:1769077833.192806:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077833.192808:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077833.192811:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077833.192820:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077833/real 1769077833] req@ffff995b84641500 x1855007616804352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077833.192821:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616804864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077833.192844:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077833.192848:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077833.192850:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077833.192854:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4818 00000100:00100000:2.0:1769077833.192857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616804352/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077833.192858:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077833.192866:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077833.192872:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077833.192876:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077833.192881:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616804992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077833.192882:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077833.192890:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616804352:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077833.192896:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077833.192901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077833.192902:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077833.192905:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4818 00000100:00080000:0.0:1769077833.192908:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077833.192910:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077833.192913:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077833.192918:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616805120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077833.192926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616804480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077833.192956:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077833.192976:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077833/real 1769077833] req@ffff995b84641180 x1855007616804480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077833.192986:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616804480/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077833.192992:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077833.192994:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077833.192997:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616804480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077833.193001:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616804608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077833.193015:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077833.193021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077833/real 1769077833] req@ffff995b84643800 x1855007616804608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077833.193029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616804608/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077833.193036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077833.193044:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077833.193046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616804608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077833.193050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616804736:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077833.193062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077833.193068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077833/real 1769077833] req@ffff995b84643480 x1855007616804736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077833.193075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616804736/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077833.193083:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077833.193087:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077833.193091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616804736:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077833.193097:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616804864:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077833.193116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077833.193128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077833/real 1769077833] req@ffff995b84642d80 x1855007616804864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077833.193141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616804864/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077833.193150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077833.193152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077833.193155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616804864:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077833.193159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616804992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077833.193183:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077833.193190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077833/real 1769077833] req@ffff995b84643100 x1855007616804992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077833.193198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616804992/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077833.193204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077833.193206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077833.193209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616804992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077833.193223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616805120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077833.193234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077833.193240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077833/real 1769077833] req@ffff995b84643b80 x1855007616805120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077833.193247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616805120/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077833.193252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077833.193254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077833.193256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616805120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077833.896552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077833.896647:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077834.216575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077834.216576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077834.216596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077834.216631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077834.920560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077834.920621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077835.240528:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077835.240544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077835.240552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077835.240556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077835.944562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077835.944621:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077836.264567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077836.264567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077836.264597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077836.264645:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077836.968562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077836.968655:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077837.288569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077837.288569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077837.288589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077837.288591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077837.992568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077837.992620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077838.312538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077838.312545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077838.312553:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077838.312561:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077838.312566:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077838.312577:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4824 00000100:00080000:0.0:1769077838.312590:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077838.312593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077838.312601:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077838.312606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077838.312646:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616805248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077838.312668:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077838.312671:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077838.312673:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077838.312676:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4824 00000100:00080000:0.0:1769077838.312680:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077838.312684:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077838.312692:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077838.312701:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616805248:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077838.312708:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616805376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077838.312722:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077838.312726:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077838.312729:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077838.312733:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4824 00000100:00080000:0.0:1769077838.312738:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077838.312743:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077838.312749:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077838.312762:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616805504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077838.312765:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077838.312775:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077838.312778:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077838.312779:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077838.312784:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4824 00000100:00080000:0.0:1769077838.312787:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077838.312790:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077838.312794:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077838.312802:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616805632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077838.312808:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077838.312812:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769077838.312812:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077838/real 1769077838] req@ffff995b84642a00 x1855007616805248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077838.312815:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077838.312820:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4824 00000100:00080000:0.0:1769077838.312823:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077838.312829:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077838.312835:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077838.312846:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616805760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077838.312854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077838.312857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616805248/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077838.312858:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077838.312861:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077838.312870:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4824 00000100:00080000:2.0:1769077838.312873:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077838.312874:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077838.312882:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077838.312883:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077838.312890:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616805248:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077838.312893:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077838.312907:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616805888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077838.312912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616805376:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077838.312938:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077838.312939:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077838.312946:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077838.312949:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077838.312954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077838/real 1769077838] req@ffff995b84640700 x1855007616805376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077838.312958:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4824 00000100:00080000:0.0:1769077838.312967:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077838.312972:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616805376/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077838.312977:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077838.312985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769077838.312989:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077838.312990:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077838.312995:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616805376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077838.313001:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616805504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077838.313011:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616806016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077838.313017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077838.313027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077838/real 1769077838] req@ffff995b84641880 x1855007616805504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077838.313037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616805504/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077838.313049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077838.313052:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077838.313054:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616805504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077838.313058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616805632:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077838.313068:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077838.313074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077838/real 1769077838] req@ffff995b84643b80 x1855007616805632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077838.313082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616805632/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077838.313087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077838.313089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077838.313091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616805632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077838.313094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616805760:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077838.313104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077838.313109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077838/real 1769077838] req@ffff995b84643100 x1855007616805760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077838.313117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616805760/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077838.313122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077838.313124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077838.313126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616805760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077838.313138:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616805888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077838.313147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077838.313153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077838/real 1769077838] req@ffff995b84642d80 x1855007616805888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077838.313160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616805888/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077838.313166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077838.313167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077838.313169:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616805888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077838.313173:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616806016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077838.313182:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077838.313188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077838/real 1769077838] req@ffff995b84643480 x1855007616806016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077838.313211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616806016/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077838.313218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077838.313220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077838.313222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616806016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077839.016572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077839.016657:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077839.336578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077839.336578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077839.336584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077839.336616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077840.040592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077840.040664:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077840.360587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077840.360635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077840.360635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077840.360636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077841.064540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077841.064599:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077841.384558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077841.384563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077841.384566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077841.384707:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077842.088571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077842.088640:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077842.408521:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077842.408538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077842.408656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077842.408685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077843.112544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077843.112597:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077843.432607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077843.432608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077843.432613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077843.432613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077843.432676:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077843.432684:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077843.432689:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077843.432703:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4829 00000100:00080000:0.0:1769077843.432718:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077843.432725:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077843.432764:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616806144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077843.432790:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077843.432793:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077843.432795:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077843.432798:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4829 00000100:00080000:0.0:1769077843.432800:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077843.432804:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077843.432812:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077843.432821:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616806272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077843.432830:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077843.432832:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616806144:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077843.432833:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077843.432837:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077843.432845:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4829 00000100:00080000:0.0:1769077843.432847:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077843.432849:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077843.432853:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077843.432858:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616806400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077843.432865:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077843.432868:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077843.432869:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077843.432872:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4829 00000100:00080000:0.0:1769077843.432874:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077843.432876:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077843.432880:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077843.432890:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616806528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077843.432895:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077843.432898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077843.432900:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077843.432902:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4829 00000100:00080000:0.0:1769077843.432905:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077843.432907:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077843.432910:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077843.432915:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077843.432922:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616806656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077843.432937:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077843.432940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077843.432941:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077843.432944:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4829 00000100:00080000:0.0:1769077843.432947:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077843.432953:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077843.432954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077843/real 1769077843] req@ffff995b84643800 x1855007616806144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077843.432960:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077843.432972:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616806784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077843.432977:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616806144/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077843.432986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077843.432995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077843.432997:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077843.432998:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077843.433002:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4829 00000100:00080000:0.0:1769077843.433005:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077843.433006:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077843.433011:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077843.433014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616806144:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077843.433016:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077843.433027:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616806912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077843.433042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616806272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077843.433060:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077843.433069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077843/real 1769077843] req@ffff995b84641180 x1855007616806272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077843.433078:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616806272/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077843.433083:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077843.433086:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077843.433096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616806272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077843.433100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616806400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077843.433127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077843.433136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077843/real 1769077843] req@ffff995b84641500 x1855007616806400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077843.433144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616806400/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077843.433161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077843.433164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077843.433166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616806400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077843.433170:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616806528:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077843.433182:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077843.433188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077843/real 1769077843] req@ffff995b84640000 x1855007616806528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077843.433195:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616806528/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077843.433201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077843.433203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077843.433204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616806528:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077843.433208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616806656:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077843.433218:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077843.433223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077843/real 1769077843] req@ffff995b84640380 x1855007616806656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077843.433230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616806656/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077843.433236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077843.433238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077843.433240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616806656:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077843.433243:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616806784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077843.433252:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077843.433258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077843/real 1769077843] req@ffff995b84641f80 x1855007616806784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077843.433265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616806784/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077843.433270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077843.433272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077843.433274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616806784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077843.433285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616806912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077843.433301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077843.433312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077843/real 1769077843] req@ffff995b84640a80 x1855007616806912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077843.433331:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616806912/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077843.433340:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077843.433343:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077843.433346:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616806912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077844.136547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077844.136604:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077844.456613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077844.456643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077844.456668:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077844.456734:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077845.160529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077845.160595:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077845.480583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077845.480583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077845.480587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077845.480641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077846.184519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077846.184586:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077846.504561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077846.504561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077846.504609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077846.504616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077847.208510:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077847.208648:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077847.528575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077847.528576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077847.528590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077847.528616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077848.232552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077848.232612:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077848.552559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077848.552561:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077848.552563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077848.552563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077848.552572:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077848.552577:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077848.552586:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4834 00000100:00100000:3.0:1769077848.552593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077848.552598:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077848.552606:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077848.552645:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616807040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077848.552672:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077848.552675:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077848.552676:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077848.552678:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4834 00000100:00080000:0.0:1769077848.552680:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077848.552683:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077848.552692:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077848.552700:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616807168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077848.552709:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616807040:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077848.552710:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077848.552715:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077848.552718:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077848.552723:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4834 00000100:00080000:0.0:1769077848.552725:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077848.552727:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077848.552730:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077848.552738:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616807296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077848.552743:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077848.552745:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077848.552747:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077848.552749:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4834 00000100:00080000:0.0:1769077848.552752:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077848.552755:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077848.552758:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077848.552766:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616807424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077848.552771:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077848.552774:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077848.552792:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077848.552795:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4834 00000100:00080000:0.0:1769077848.552798:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077848.552802:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769077848.552807:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077848.552810:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077848.552821:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616807552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077848.552829:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077848.552832:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077848.552834:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077848.552837:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4834 00000100:00080000:0.0:1769077848.552840:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077848.552843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077848.552847:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077848.552852:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077848/real 1769077848] req@ffff995b84642680 x1855007616807040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077848.552857:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616807680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077848.552902:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077848.552905:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077848.552907:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077848.552928:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4834 00000100:00100000:2.0:1769077848.552929:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616807040/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077848.552930:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077848.552936:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077848.552941:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077848.552951:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616807808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077848.552952:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077848.552963:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077848.552969:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616807040:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077848.553005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616807168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077848.553047:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077848.553063:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077848/real 1769077848] req@ffff995b84640e00 x1855007616807168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077848.553078:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616807168/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077848.553085:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077848.553088:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077848.553091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616807168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077848.553096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616807296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077848.553135:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077848.553148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077848/real 1769077848] req@ffff995b84641c00 x1855007616807296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077848.553162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616807296/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077848.553191:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077848.553194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077848.553198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616807296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077848.553204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616807424:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077848.553225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077848.553238:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077848/real 1769077848] req@ffff995b84640a80 x1855007616807424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077848.553250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616807424/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077848.553260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077848.553263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077848.553267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616807424:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077848.553273:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616807552:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077848.553289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077848.553297:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077848/real 1769077848] req@ffff995b84641f80 x1855007616807552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077848.553305:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616807552/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077848.553311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077848.553314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077848.553316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616807552:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077848.553319:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616807680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077848.553330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077848.553336:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077848/real 1769077848] req@ffff995b84640380 x1855007616807680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077848.553343:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616807680/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077848.553353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077848.553356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077848.553360:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616807680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077848.553365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616807808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077848.553386:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077848.553398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077848/real 1769077848] req@ffff995b84640000 x1855007616807808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077848.553411:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616807808/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077848.553420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077848.553423:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077848.553426:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616807808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077849.256518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077849.256637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077849.576575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077849.576575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077849.576591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077849.576598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077850.280526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077850.280577:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077850.600587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077850.600587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077850.600587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077850.600588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077851.304568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077851.304658:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077851.624551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077851.624551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077851.624577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077851.624580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077852.328550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077852.328619:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077852.648561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077852.648561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077852.648566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077852.648590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077853.352602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077853.352690:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077853.672574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077853.672598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077853.672599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077853.672641:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077853.672648:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077853.672664:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077853.672676:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4839 00000100:00100000:0.0:1769077853.672683:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077853.672692:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077853.672703:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077853.672742:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616807936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077853.672783:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077853.672787:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077853.672789:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077853.672794:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4839 00000100:00080000:3.0:1769077853.672810:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077853.672815:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077853.672836:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077853.672851:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616807936:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077853.672852:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616808064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077853.672874:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077853.672878:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077853.672882:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077853.672890:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4839 00000100:00080000:3.0:1769077853.672897:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077853.672901:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077853.672907:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077853.672918:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616808192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077853.672929:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077853.672932:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077853.672934:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769077853.672954:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077853.672959:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4839 00000100:00080000:3.0:1769077853.672964:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077853.672969:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077853.672975:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077853.673010:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077853/real 1769077853] req@ffff995b8455a300 x1855007616807936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077853.673041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616807936/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077853.673051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077853.673057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077853.673062:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616807936:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077853.673094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077853.673094:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616808320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077853.673119:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077853.673125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077853.673125:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077853.673129:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077853.673135:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4839 00000100:00080000:3.0:1769077853.673141:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077853.673144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077853/real 1769077853] req@ffff995b84559c00 x1855007616808064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077853.673148:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077853.673157:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077853.673163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616808064/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077853.673172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077853.673175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077853.673177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077853.673183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077853.673199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077853.673207:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077853/real 1769077853] req@ffff995b84559880 x1855007616808192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077853.673215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616808192/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077853.673223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077853.673226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077853.673228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077853.673240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808320:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077853.673257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077853.673266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077853/real 1769077853] req@ffff995b84558000 x1855007616808320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077853.673277:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616808320/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077853.673284:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077853.673287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077853.673289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808320:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077853.673325:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616808448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077853.673392:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077853.673397:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808448:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077853.673400:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077853.673407:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077853.673414:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4839 00000100:00080000:3.0:1769077853.673433:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077853.673439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077853.673446:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077853.673494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077853/real 1769077853] req@ffff995b84559500 x1855007616808448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077853.673498:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077853.673513:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616808448/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077853.673527:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616808576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077853.673539:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077853.673546:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077853.673547:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077853.673552:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808448:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769077853.673556:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077853.673561:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077853.673569:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4839 00000100:00100000:0.0:1769077853.673571:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616808576:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077853.673579:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077853.673587:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077853.673594:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077853.673599:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077853.673611:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616808704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077853.673613:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077853/real 1769077853] req@ffff995b8455bb80 x1855007616808576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077853.673652:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616808576/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077853.673668:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077853.673671:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077853.673675:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616808576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077853.673684:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616808704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077853.673697:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077853.673704:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077853/real 1769077853] req@ffff995b84558a80 x1855007616808704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077853.673713:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616808704/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077853.673720:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077853.673722:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077853.673725:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616808704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077854.376584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077854.376651:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077854.696582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077854.696585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077854.696610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077854.696656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077855.400592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077855.400694:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077855.720557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077855.720562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077855.720564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077855.720581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077856.424545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077856.424686:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077856.744592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077856.744593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077856.744593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077856.744605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077857.448586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077857.448651:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077857.768562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077857.768567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077857.768568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077857.768569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077858.472579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077858.472660:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077858.792588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077858.792588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077858.792593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077858.792608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077858.792628:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077858.792633:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077858.792637:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077858.792646:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4844 00000100:00080000:3.0:1769077858.792657:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077858.792664:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077858.792709:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616808832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077858.792739:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077858.792742:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077858.792744:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077858.792747:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4844 00000100:00080000:3.0:1769077858.792750:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077858.792754:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077858.792763:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077858.792776:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616808960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077858.792784:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616808832:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077858.792787:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077858.792791:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077858.792793:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077858.792800:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4844 00000100:00080000:3.0:1769077858.792805:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077858.792809:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077858.792815:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077858.792832:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616809088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077858.792843:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077858.792846:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077858.792849:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077858.792881:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4844 00000100:00080000:3.0:1769077858.792886:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077858.792891:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077858.792897:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769077858.792899:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077858.792922:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616809216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077858.792932:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077858.792934:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077858.792936:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077858.792941:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4844 00000100:00080000:3.0:1769077858.792946:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077858.792951:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077858.792957:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077858.792959:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077858/real 1769077858] req@ffff995b8455aa00 x1855007616808832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077858.792974:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616809344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077858.792985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616808832/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077858.792987:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077858.792998:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077858.793002:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077858.793009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077858.793010:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4844 00000100:00080000:3.0:1769077858.793018:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077858.793019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077858.793025:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077858.793029:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616808832:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769077858.793033:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077858.793051:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616809472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077858.793062:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077858.793064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808960:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769077858.793071:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077858.793074:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077858.793080:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4844 00000100:00080000:3.0:1769077858.793085:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077858.793088:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077858.793092:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077858.793104:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616809600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077858.793117:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077858.793136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077858/real 1769077858] req@ffff995b8455b100 x1855007616808960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077858.793152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616808960/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077858.793162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077858.793166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077858.793170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616808960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077858.793177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077858.793198:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077858.793211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077858/real 1769077858] req@ffff995b8455a680 x1855007616809088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077858.793224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616809088/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077858.793243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077858.793247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077858.793250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077858.793257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809216:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077858.793280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077858.793291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077858/real 1769077858] req@ffff995b84558a80 x1855007616809216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077858.793299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616809216/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077858.793306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077858.793308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077858.793310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809216:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077858.793314:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809344:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077858.793327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077858.793333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077858/real 1769077858] req@ffff995b8455bb80 x1855007616809344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077858.793341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616809344/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077858.793346:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077858.793348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077858.793351:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809344:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077858.793367:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616809472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077858.793378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077858.793385:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077858/real 1769077858] req@ffff995b84559500 x1855007616809472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077858.793392:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616809472/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077858.793398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077858.793400:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077858.793403:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616809472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077858.793406:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616809600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077858.793415:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077858.793420:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077858/real 1769077858] req@ffff995b84558000 x1855007616809600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077858.793427:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616809600/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077858.793433:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077858.793435:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077858.793437:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616809600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077859.496601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077859.496691:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077859.816562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077859.816562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077859.816563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077859.816568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077860.520585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077860.520654:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077860.840575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077860.840576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077860.840585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077860.840633:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077861.544567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077861.544635:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077861.864548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077861.864549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077861.864561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077861.864587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077862.568538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077862.568644:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077862.888604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077862.888615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077862.888650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077862.888672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077863.592571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077863.592644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077863.912508:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077863.912549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077863.912550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077863.912564:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077863.912573:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077863.912579:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077863.912580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077863.912591:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4849 00000100:00080000:2.0:1769077863.912603:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077863.912611:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077863.912657:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616809728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077863.912685:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077863.912688:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077863.912689:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077863.912693:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4849 00000100:00080000:2.0:1769077863.912696:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077863.912701:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077863.912709:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077863.912716:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616809728:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077863.912722:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616809856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077863.912731:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077863.912733:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077863.912735:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077863.912738:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4849 00000100:00080000:2.0:1769077863.912740:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077863.912742:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077863.912745:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077863.912754:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616809984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077863.912760:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077863.912763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077863.912764:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077863.912783:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4849 00000100:00080000:2.0:1769077863.912785:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077863.912788:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077863.912792:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077863.912801:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616810112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077863.912807:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077863.912810:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077863.912811:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077863.912814:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4849 00000100:00080000:2.0:1769077863.912817:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077863.912819:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077863.912822:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077863.912830:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616810240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077863.912837:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077863.912840:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077863.912841:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077863.912845:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4849 00000100:00080000:2.0:1769077863.912847:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077863.912851:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077863.912854:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809856:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077863.912855:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077863.912868:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616810368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077863.912875:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077863.912877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077863.912878:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077863.912882:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4849 00000100:00080000:2.0:1769077863.912885:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077863.912888:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077863.912892:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077863.912897:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077863.912907:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616810496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077863.912918:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616810112:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077863.912940:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616810240:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077863.912955:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616810368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077863.912969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616810496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077864.616539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077864.616604:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077864.936554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077864.936660:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077864.936665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077864.936677:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077865.640570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077865.640646:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077865.960575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077865.960579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077865.960613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077865.960616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077866.664516:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077866.664586:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077866.984553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077866.984565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077866.984589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077866.984668:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:3.0:1769077866.984772:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:3.0:1769077866.984816:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769077866.984838:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077866.984884:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077866.984896:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077866.984908:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077866.984921:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769077866.984932:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769077866.984933:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077863/real 1769077866] req@ffff995b85aca300 x1855007616809856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1769077866.984945:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769077866.984973:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616809856/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077866.984986:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077866.984993:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077866.984998:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077866.985010:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077863/real 1769077866] req@ffff995b85ac8380 x1855007616809984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077866.985040:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616809984/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077866.985052:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077866.985057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077866.985061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616809984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077866.985069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077863/real 1769077866] req@ffff995b85acb100 x1855007616810112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077866.985082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616810112/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077866.985091:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077866.985096:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077866.985101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616810112:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077866.985109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077863/real 1769077866] req@ffff995b85ac9c00 x1855007616810240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077866.985122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616810240/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077866.985131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077866.985134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077866.985138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616810240:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077866.985147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077863/real 1769077866] req@ffff995b85acaa00 x1855007616809728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077866.985160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616809728/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077866.985169:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077866.985173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077866.985177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616809728:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077866.985217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077863/real 1769077866] req@ffff995b85ac9880 x1855007616810368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077866.985231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616810368/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077866.985239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077866.985242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077866.985245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616810368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077866.985253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077863/real 1769077866] req@ffff995b85acbb80 x1855007616810496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077866.985265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616810496/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077866.985272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077866.985276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077866.985279:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616810496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077867.688549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077867.688615:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077868.008577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077868.008577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077868.008586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077868.008607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077868.712559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077868.712634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077869.032558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077869.032558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077869.032559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077869.032611:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077869.032615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077869.032617:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077869.032622:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077869.032630:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4854 00000100:00080000:3.0:1769077869.032641:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077869.032648:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077869.032705:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616810624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077869.032726:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077869.032729:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077869.032731:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077869.032734:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4854 00000100:00080000:3.0:1769077869.032737:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077869.032741:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077869.032749:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077869.032760:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616810624:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077869.032761:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616810752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077869.032772:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077869.032774:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077869.032776:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077869.032778:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4854 00000100:00080000:3.0:1769077869.032781:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077869.032784:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077869.032789:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077869.032801:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616810880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077869.032809:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077869.032812:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077869.032814:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077869.032818:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4854 00000100:00080000:3.0:1769077869.032822:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077869.032826:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077869.032831:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077869.032843:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616811008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077869.032859:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077869.032862:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1769077869.032863:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077869.032865:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077869.032873:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4854 00000100:00080000:3.0:1769077869.032877:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077869.032882:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077869.032887:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077869.032900:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616811136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077869.032909:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077869.032914:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769077869.032916:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077869/real 1769077869] req@ffff995b84559880 x1855007616810624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077869.032918:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077869.032928:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4854 00000100:00080000:3.0:1769077869.032935:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077869.032941:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077869.032947:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077869.032949:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616810624/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077869.032964:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077869.032964:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616811264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077869.032980:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077869.032982:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769077869.032987:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769077869.032992:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616810624:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077869.032992:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077869.033005:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4854 00000100:00080000:3.0:1769077869.033013:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077869.033017:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077869.033025:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077869.033032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616810752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077869.033040:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616811392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077869.033067:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077869.033085:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077869/real 1769077869] req@ffff995b84559c00 x1855007616810752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077869.033098:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616810752/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077869.033107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077869.033111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077869.033114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616810752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077869.033122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616810880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077869.033145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077869.033156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077869/real 1769077869] req@ffff995b8455a300 x1855007616810880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077869.033169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616810880/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077869.033179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077869.033183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077869.033187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616810880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077869.033192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811008:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077869.033213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077869.033225:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077869/real 1769077869] req@ffff995b8455ad80 x1855007616811008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077869.033237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616811008/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077869.033248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077869.033251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077869.033254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077869.033261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811136:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077869.033280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077869.033291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077869/real 1769077869] req@ffff995b8455b480 x1855007616811136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077869.033304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616811136/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077869.033313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077869.033316:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077869.033319:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811136:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077869.033325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616811264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077869.033342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077869.033352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077869/real 1769077869] req@ffff995b84558700 x1855007616811264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077869.033364:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616811264/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077869.033371:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077869.033375:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077869.033378:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616811264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077869.033401:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616811392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077869.033420:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077869.033433:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077869/real 1769077869] req@ffff995b84559880 x1855007616811392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077869.033445:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616811392/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077869.033486:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077869.033490:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077869.033498:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616811392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077869.736596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077869.736703:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077870.056570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077870.056582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077870.056603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077870.056773:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077870.760592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077870.760676:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077871.080556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077871.080557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077871.080566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077871.080575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077871.784590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077871.784696:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077872.104606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077872.104613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077872.104615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077872.104615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077872.808561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077872.808656:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077873.128567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077873.128567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077873.128569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077873.128587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769077873.832661:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077873.832726:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077874.152586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077874.152586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077874.152591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077874.152661:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077874.152668:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769077874.152671:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077874.152673:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077874.152683:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4859 00000100:00080000:2.0:1769077874.152693:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077874.152700:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077874.152739:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616811520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077874.152760:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077874.152763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077874.152765:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077874.152768:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4859 00000100:00080000:2.0:1769077874.152770:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077874.152773:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077874.152782:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077874.152793:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616811520:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077874.152793:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616811648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077874.152806:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077874.152808:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077874.152810:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077874.152813:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4859 00000100:00080000:2.0:1769077874.152816:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077874.152818:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077874.152821:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077874.152836:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616811776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077874.152842:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077874.152844:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077874.152845:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077874.152848:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4859 00000100:00080000:2.0:1769077874.152851:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077874.152854:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077874.152857:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769077874.152864:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077874.152868:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616811904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077874.152876:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077874.152878:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077874.152880:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077874.152883:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4859 00000100:00080000:2.0:1769077874.152886:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077874.152889:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077874.152892:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077874.152904:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616812032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077874.152910:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077874/real 1769077874] req@ffff995b85acb800 x1855007616811520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077874.152919:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077874.152925:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077874.152928:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077874.152932:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616811520/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077874.152937:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4859 00000100:00080000:2.0:1769077874.152947:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077874.152948:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077874.152957:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077874.152957:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077874.152964:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616811520:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077874.152966:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077874.152979:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616812160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077874.152987:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811648:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077874.153002:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077874.153005:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077874.153007:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077874.153011:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4859 00000100:00080000:2.0:1769077874.153015:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077874.153016:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077874.153022:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077874.153027:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077874.153031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077874/real 1769077874] req@ffff995b85ac8a80 x1855007616811648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077874.153037:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616812288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077874.153079:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616811648/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077874.153086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077874.153088:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077874.153091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077874.153096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077874.153109:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077874.153116:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077874/real 1769077874] req@ffff995b85acbb80 x1855007616811776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077874.153124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616811776/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077874.153132:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077874.153133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077874.153135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077874.153139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811904:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077874.153150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077874.153156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077874/real 1769077874] req@ffff995b85ac9880 x1855007616811904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077874.153163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616811904/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077874.153173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077874.153175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077874.153177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616811904:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077874.153180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812032:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077874.153189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077874.153195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077874/real 1769077874] req@ffff995b85acaa00 x1855007616812032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077874.153202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616812032/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077874.153208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077874.153210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077874.153211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812032:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077874.153223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616812160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077874.153233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077874.153239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077874/real 1769077874] req@ffff995b85ac9c00 x1855007616812160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077874.153246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616812160/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077874.153251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077874.153253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077874.153256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616812160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077874.153260:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616812288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077874.153269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077874.153276:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077874/real 1769077874] req@ffff995b85acb100 x1855007616812288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077874.153295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616812288/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077874.153305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077874.153308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077874.153312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616812288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077874.856531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077874.856593:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077875.176568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077875.176610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077875.176650:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077875.176685:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077875.880540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077875.880608:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077876.200615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077876.200703:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077876.200828:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077876.201498:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077876.904597:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077876.904705:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077877.224553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077877.224556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077877.224619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077877.224661:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077877.928566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077877.928603:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077878.248543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077878.248560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077878.248572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077878.248588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077878.952575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077878.952633:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077879.272595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077879.272684:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077879.272691:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077879.272696:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077879.272718:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4864 00000100:00080000:0.0:1769077879.272730:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077879.272737:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077879.272774:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616812416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077879.272845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616812416:192.168.204.132@tcp:250:kworker.0 00000800:00000100:2.0:1769077879.272912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077879.273010:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077879.273014:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077879.273017:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077879.273021:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4864 00000100:00080000:0.0:1769077879.273025:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077879.273029:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077879.273039:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077879.273052:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616812544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077879.273149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077879/real 1769077879] req@ffff995b84641500 x1855007616812416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077879.273166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616812416/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077879.273177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077879.273182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077879.273186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616812416:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077879.273208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077879.273230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077879.273254:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077879/real 1769077879] req@ffff995b84641180 x1855007616812544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077879.273275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616812544/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077879.273283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077879.273286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077879.273289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812544:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077879.273394:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077879.273398:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077879.273400:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077879.273405:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4864 00000100:00080000:0.0:1769077879.273409:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077879.273413:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077879.273419:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077879.273432:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616812672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077879.273442:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077879.273445:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077879.273447:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077879.273450:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4864 00000100:00080000:0.0:1769077879.273485:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077879.273488:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077879.273493:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077879.273503:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616812800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077879.273524:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077879.273527:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077879.273528:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077879.273532:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4864 00000100:00080000:0.0:1769077879.273535:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077879.273538:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077879.273542:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077879.273551:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616812928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077879.273559:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077879.273562:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077879.273564:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077879.273568:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4864 00000100:00080000:0.0:1769077879.273570:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077879.273574:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077879.273577:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077879.273586:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616813056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077879.273594:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077879.273597:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077879.273599:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077879.273602:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4864 00000100:00080000:0.0:1769077879.273605:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077879.273608:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077879.273612:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077879.273857:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077879.273889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077879.273896:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000800:00000100:2.0:1769077879.273980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077879.273994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077879/real 1769077879] req@ffff995b84643800 x1855007616812672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077879.274020:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616813184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077879.274076:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077879.274157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616812672/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077879.274168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077879.274172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077879.274175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077879.274182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812800:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077879.274206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077879.274218:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077879/real 1769077879] req@ffff995b84643480 x1855007616812800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077879.274229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616812800/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077879.274247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077879.274251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077879.274254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077879.274259:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812928:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077879.274278:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077879.274288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077879/real 1769077879] req@ffff995b84642d80 x1855007616812928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077879.274299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616812928/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077879.274308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077879.274311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077879.274314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616812928:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077879.274320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616813056:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077879.274337:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077879.274348:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077879/real 1769077879] req@ffff995b84643100 x1855007616813056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077879.274359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616813056/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077879.274379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077879.274383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077879.274387:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616813056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077879.274410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616813184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077879.274432:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077879.274443:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077879/real 1769077879] req@ffff995b84643b80 x1855007616813184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077879.274472:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616813184/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077879.274493:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077879.274497:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077879.274500:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616813184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077879.976554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077879.977482:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077880.296542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077880.296543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077880.296549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077880.296658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077881.000575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077881.000642:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077881.320555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077881.320568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077881.320711:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077881.320741:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077882.024559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077882.024613:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077882.344568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077882.344569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077882.344585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077882.344625:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077883.048525:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077883.048584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077883.368536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077883.368562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077883.368563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077883.368564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769077884.072612:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077884.072625:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077884.392574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077884.392574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077884.392711:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077884.392718:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077884.392722:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077884.392737:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4870 00000100:00080000:0.0:1769077884.392746:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077884.392751:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077884.392785:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616813312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077884.392805:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077884.392808:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077884.392809:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077884.392812:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4870 00000100:00080000:0.0:1769077884.392815:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077884.392819:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077884.392826:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077884.392834:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616813312:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077884.392836:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616813440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077884.392846:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077884.392848:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077884.392849:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077884.392851:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4870 00000100:00080000:0.0:1769077884.392854:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077884.392856:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077884.392858:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077884.392865:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616813568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077884.392874:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077884.392878:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077884.392880:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077884.392883:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4870 00000100:00080000:0.0:1769077884.392887:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077884.392890:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769077884.392895:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077884.392899:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077884.392915:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616813696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077884.392930:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077884.392930:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077884/real 1769077884] req@ffff995b84641880 x1855007616813312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077884.392937:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077884.392939:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077884.392947:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4870 00000100:00080000:0.0:1769077884.392950:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077884.392953:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077884.392957:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077884.392958:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616813312/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077884.392967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077884.392969:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616813824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077884.392973:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077884.392981:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077884.392981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616813312:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077884.392986:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077884.392990:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077884.392994:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4870 00000100:00080000:0.0:1769077884.392995:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077884.392998:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077884.393001:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077884.393006:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616813952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077884.393013:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077884.393014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616813440:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077884.393017:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077884.393021:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077884.393025:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4870 00000100:00080000:0.0:1769077884.393028:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077884.393030:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077884.393033:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077884.393041:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616814080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077884.393050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077884.393066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077884/real 1769077884] req@ffff995b84640700 x1855007616813440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077884.393078:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616813440/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077884.393086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077884.393089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077884.393093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616813440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077884.393100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616813568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077884.393120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077884.393131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077884/real 1769077884] req@ffff995b84642a00 x1855007616813568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077884.393134:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077884.393145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616813568/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077884.393153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077884.393156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077884.393158:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616813568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077884.393161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616813696:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077884.393173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077884.393181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077884/real 1769077884] req@ffff995b84643b80 x1855007616813696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077884.393188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616813696/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077884.393194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077884.393195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077884.393199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616813696:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077884.393202:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616813824:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077884.393212:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000800:00000100:1.0:1769077884.393219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077884.393232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077884/real 1769077884] req@ffff995b84643100 x1855007616813824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077884.393244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616813824/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077884.393253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077884.393256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077884.393259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616813824:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077884.393275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616813952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077884.393289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077884.393295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077884/real 1769077884] req@ffff995b84642d80 x1855007616813952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077884.393302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616813952/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077884.393311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077884.393313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077884.393315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616813952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077884.393318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616814080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077884.393327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077884.393332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077884/real 1769077884] req@ffff995b84643480 x1855007616814080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077884.393338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616814080/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077884.393342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077884.393344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077884.393346:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616814080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077885.096563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077885.096607:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077885.416555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077885.416555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077885.416564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077885.416569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077886.120526:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077886.120587:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077886.440554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077886.440554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077886.440554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077886.440563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077887.144573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077887.144644:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077887.464601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077887.464601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077887.464611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077887.464617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077888.168543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077888.168604:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077888.488608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077888.488613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077888.488621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077888.488622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077889.192561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077889.192637:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077889.512548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077889.512552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077889.512562:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077889.512568:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077889.512573:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077889.512580:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4875 00000100:00080000:0.0:1769077889.512589:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077889.512595:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077889.512609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077889.512619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077889.512632:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616814208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077889.512661:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077889.512665:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077889.512668:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077889.512673:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4875 00000100:00080000:0.0:1769077889.512681:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077889.512687:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077889.512691:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616814208:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077889.512697:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077889.512712:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616814336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077889.512720:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077889.512722:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077889.512724:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077889.512727:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4875 00000100:00080000:0.0:1769077889.512730:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077889.512732:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077889.512735:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077889.512742:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616814464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077889.512749:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077889.512751:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077889.512754:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769077889.512755:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077889.512759:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4875 00000100:00080000:0.0:1769077889.512763:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077889.512766:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077889.512770:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077889.512780:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616814592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077889.512787:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077889.512790:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077889.512793:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077889.512793:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077889/real 1769077889] req@ffff995b84643800 x1855007616814208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077889.512802:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4875 00000100:00080000:0.0:1769077889.512807:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077889.512811:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077889.512815:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077889.512820:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616814208/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077889.512825:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616814720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077889.512836:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077889.512837:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077889.512842:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077889.512844:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077889.512846:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077889.512849:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4875 00000100:00080000:0.0:1769077889.512854:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077889.512857:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616814208:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077889.512859:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077889.512865:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077889.512872:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616814848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077889.512877:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077889.512879:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077889.512879:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077889.512881:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4875 00000100:00080000:0.0:1769077889.512883:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077889.512887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616814336:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077889.512888:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077889.512896:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077889.512905:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616814976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077889.512917:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077889.512931:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077889/real 1769077889] req@ffff995b84641180 x1855007616814336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077889.512947:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616814336/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077889.512965:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077889.512971:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077889.512979:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616814336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077889.512988:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616814464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077889.513010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077889.513020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077889/real 1769077889] req@ffff995b84641500 x1855007616814464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077889.513027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616814464/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077889.513035:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077889.513037:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077889.513039:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616814464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077889.513044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616814592:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077889.513057:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077889.513065:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077889/real 1769077889] req@ffff995b84640000 x1855007616814592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077889.513076:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616814592/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077889.513084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077889.513088:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077889.513092:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616814592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077889.513099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616814720:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077889.513113:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077889.513119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077889/real 1769077889] req@ffff995b84640380 x1855007616814720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077889.513127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616814720/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077889.513136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077889.513140:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077889.513142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616814720:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077889.513148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616814848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077889.513162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077889.513174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077889/real 1769077889] req@ffff995b84641f80 x1855007616814848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077889.513183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616814848/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077889.513191:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077889.513194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077889.513196:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616814848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077889.513211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616814976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077889.513222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077889.513230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077889/real 1769077889] req@ffff995b84640a80 x1855007616814976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077889.513240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616814976/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077889.513244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077889.513247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077889.513250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616814976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077890.216565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077890.216621:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077890.536578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077890.536584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077890.536588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077890.536619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077891.240545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077891.240617:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077891.560572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077891.560572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077891.560577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077891.560591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077892.264531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077892.264624:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077892.584546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077892.584570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077892.584596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077892.584602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077893.288530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077893.288588:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077893.608559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077893.608560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077893.608564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077893.608601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077894.312552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077894.312617:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077894.632535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077894.632535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077894.632569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077894.632577:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077894.632581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077894.632586:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077894.632593:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077894.632604:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4880 00000100:00080000:0.0:1769077894.632615:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077894.632633:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077894.632675:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616815104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077894.632705:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077894.632709:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077894.632711:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077894.632715:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4880 00000100:00080000:0.0:1769077894.632719:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077894.632729:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077894.632732:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616815104:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077894.632742:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077894.632755:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616815232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077894.632768:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077894.632770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077894.632772:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077894.632774:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4880 00000100:00080000:0.0:1769077894.632777:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077894.632779:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077894.632787:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077894.632793:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077894.632797:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616815360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077894.632805:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077894.632808:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077894.632809:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077894.632812:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4880 00000100:00080000:0.0:1769077894.632814:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077894.632816:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077894.632820:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077894.632830:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077894/real 1769077894] req@ffff995b84641c00 x1855007616815104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077894.632831:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616815488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077894.632846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077894.632849:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616815104/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077894.632850:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077894.632854:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077894.632862:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4880 00000100:00080000:0.0:1769077894.632868:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077894.632868:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077894.632873:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077894.632877:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769077894.632878:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077894.632884:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616815104:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077894.632889:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616815616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077894.632897:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077894.632899:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077894.632901:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077894.632905:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4880 00000100:00100000:2.0:1769077894.632909:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616815232:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077894.632910:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077894.632918:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077894.632924:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077894.632931:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077894.632940:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077894/real 1769077894] req@ffff995b84640e00 x1855007616815232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077894.632951:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616815744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077894.632952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616815232/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077894.632972:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077894.632978:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077894.632982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077894.632982:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616815232:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077894.632992:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769077894.632993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616815360:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077894.632996:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077894.633002:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4880 00000100:00080000:0.0:1769077894.633008:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077894.633012:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077894.633020:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077894.633021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077894/real 1769077894] req@ffff995b84642680 x1855007616815360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077894.633029:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077894.633041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616815360/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077894.633048:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616815872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077894.633063:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077894.633073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077894.633081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616815360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077894.633089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616815488:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077894.633115:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077894.633127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077894/real 1769077894] req@ffff995b84640a80 x1855007616815488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077894.633136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616815488/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077894.633143:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077894.633145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077894.633148:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616815488:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077894.633152:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616815616:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077894.633165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077894.633172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077894/real 1769077894] req@ffff995b84641f80 x1855007616815616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077894.633179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616815616/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077894.633185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077894.633187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077894.633189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616815616:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077894.633203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616815744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077894.633213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077894.633228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077894/real 1769077894] req@ffff995b84640380 x1855007616815744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077894.633241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616815744/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077894.633247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077894.633275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077894.633278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616815744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077894.633282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616815872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077894.633302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077894.633313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077894/real 1769077894] req@ffff995b84640000 x1855007616815872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077894.633326:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616815872/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077894.633334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077894.633337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077894.633341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616815872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077895.336545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077895.336618:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077895.656575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077895.656575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077895.656672:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077895.656690:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077896.360544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077896.360621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077896.680545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077896.680546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077896.680874:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077896.680922:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077897.384522:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077897.384608:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077897.704563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077897.704563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077897.704563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077897.704563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077898.408601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077898.408667:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077898.728533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077898.728533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077898.728598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077898.728617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077899.432581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077899.432641:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077899.752604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077899.752624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077899.752625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077899.752637:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077899.752647:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077899.752653:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077899.752665:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4885 00000100:00100000:1.0:1769077899.752672:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077899.752682:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077899.752695:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077899.752741:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616816000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077899.752769:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077899.752772:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077899.752774:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077899.752777:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4885 00000100:00080000:0.0:1769077899.752781:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077899.752786:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077899.752796:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077899.752809:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616816000:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077899.752810:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616816128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077899.752834:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077899.752836:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077899.752837:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077899.752844:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4885 00000100:00080000:0.0:1769077899.752848:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077899.752851:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077899.752855:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077899.752869:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616816256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077899.752876:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077899.752879:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077899.752880:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077899.752884:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4885 00000100:00080000:0.0:1769077899.752888:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077899.752893:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769077899.752898:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077899.752899:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077899.752919:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616816384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077899.752931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077899.752935:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077899.752937:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077899.752940:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077899/real 1769077899] req@ffff995b84641500 x1855007616816000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077899.752946:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4885 00000100:00080000:0.0:1769077899.752959:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077899.752967:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077899.752968:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616816000/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077899.752976:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769077899.752983:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.752988:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077899.752991:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616816000:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077899.753003:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616816512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077899.753012:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077899.753015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077899.753017:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077899.753020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616816128:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077899.753027:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4885 00000100:00080000:0.0:1769077899.753035:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077899.753038:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077899.753041:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077899.753048:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077899.753055:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616816640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077899.753064:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077899/real 1769077899] req@ffff995b84641180 x1855007616816128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077899.753066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077899.753082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077899.753088:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077899.753099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616816128/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077899.753102:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4885 00000100:00080000:0.0:1769077899.753113:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077899.753123:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077899.753125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769077899.753130:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077899.753132:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077899.753142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616816128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077899.753147:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616816768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077899.753198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616816256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077899.753228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077899.753238:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077899/real 1769077899] req@ffff995b84643800 x1855007616816256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.753247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616816256/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.753255:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.753258:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077899.753260:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616816256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077899.753265:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616816384:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077899.753277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077899.753283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077899/real 1769077899] req@ffff995b84643480 x1855007616816384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.753290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616816384/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.753296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.753298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077899.753300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616816384:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077899.753303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616816512:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077899.753327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077899.753339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077899/real 1769077899] req@ffff995b84642d80 x1855007616816512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.753359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616816512/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.753366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.753368:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077899.753370:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616816512:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077899.753385:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616816640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077899.753400:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077899.753408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077899/real 1769077899] req@ffff995b84643100 x1855007616816640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.753415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616816640/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.753420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.753423:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077899.753425:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616816640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077899.753429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616816768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077899.753438:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077899.753444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077899/real 1769077899] req@ffff995b84643b80 x1855007616816768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.753450:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616816768/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.753479:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.753482:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077899.753484:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616816768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077900.456579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077900.456694:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077900.776567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077900.776584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077900.776609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077900.776643:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077901.480606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077901.480665:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077901.800568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077901.800578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077901.800588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077901.800615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077902.504587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077902.504683:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077902.824539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077902.824576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077902.824608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077902.824616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077903.528544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077903.528613:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077903.848593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077903.848593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077903.848647:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077903.848653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077904.552613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077904.552691:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077904.872586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077904.872635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077904.872642:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077904.872649:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077904.872655:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077904.872663:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077904.872666:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4890 00000100:00080000:0.0:1769077904.872683:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077904.872683:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077904.872703:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077904.872744:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616816896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077904.872770:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077904.872773:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077904.872775:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077904.872778:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4890 00000100:00080000:0.0:1769077904.872781:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077904.872784:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077904.872794:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077904.872804:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616817024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077904.872811:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077904.872814:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077904.872815:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077904.872818:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4890 00000100:00080000:0.0:1769077904.872820:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077904.872824:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077904.872827:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616816896:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769077904.872828:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077904.872840:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616817152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077904.872846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077904.872847:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077904.872848:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077904.872851:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4890 00000100:00080000:0.0:1769077904.872854:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077904.872856:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077904.872867:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077904.872879:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616817280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077904.872885:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077904.872887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077904.872888:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077904.872890:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4890 00000100:00080000:0.0:1769077904.872892:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077904.872895:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077904.872898:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077904.872905:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616817408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077904.872910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077904.872913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077904.872916:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769077904.872918:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077904.872922:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4890 00000100:00080000:0.0:1769077904.872925:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077904.872928:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077904.872931:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077904.872941:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616817536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077904.872950:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077904.872954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077904.872956:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077904.872960:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4890 00000100:00080000:0.0:1769077904.872963:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077904.872967:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077904.872973:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077904.872983:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077904/real 1769077904] req@ffff995b84642a00 x1855007616816896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077904.872988:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616817664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077904.873023:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616816896/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077904.873038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077904.873046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077904.873051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616816896:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077904.873130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077904.873159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077904.873173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077904/real 1769077904] req@ffff995b84640700 x1855007616817024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077904.873183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616817024/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077904.873190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077904.873193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077904.873196:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077904.873205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077904.873221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077904.873228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077904/real 1769077904] req@ffff995b84641880 x1855007616817152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077904.873236:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616817152/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077904.873244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077904.873246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077904.873248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077904.873252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817280:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077904.873267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077904.873278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077904/real 1769077904] req@ffff995b84643b80 x1855007616817280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077904.873294:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616817280/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077904.873304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077904.873308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077904.873312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077904.873318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817408:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077904.873336:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077904.873346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077904/real 1769077904] req@ffff995b84643100 x1855007616817408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077904.873359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616817408/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077904.873369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077904.873373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077904.873376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077904.873383:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616817536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077904.873398:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077904.873408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077904/real 1769077904] req@ffff995b84642d80 x1855007616817536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077904.873420:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616817536/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077904.873437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077904.873441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077904.873444:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616817536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077904.873450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616817664:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077904.873492:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077904.873503:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077904/real 1769077904] req@ffff995b84643480 x1855007616817664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077904.873515:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616817664/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077904.873525:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077904.873529:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077904.873532:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616817664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077905.576521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077905.576632:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077905.896576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077905.896578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077905.896727:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077905.896786:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077906.600598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077906.600675:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077906.920642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077906.920649:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077906.920671:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077906.920684:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077907.624536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077907.624606:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077907.944570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077907.944570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077907.944579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077907.944581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077908.648553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077908.648620:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077908.968596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077908.968597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077908.968598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077908.968624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077909.672558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077909.672621:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077909.992560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077909.992560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077909.992560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077909.992564:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077909.992573:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077909.992589:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077909.992600:0:38:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4895 00000100:00080000:0.0:1769077909.992613:0:38:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077909.992618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077909.992623:0:38:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077909.992670:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616817792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077909.992693:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077909.992696:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077909.992697:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077909.992701:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4895 00000100:00080000:0.0:1769077909.992704:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077909.992707:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077909.992716:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077909.992725:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616817920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077909.992733:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616817792:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077909.992735:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077909.992741:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077909.992743:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077909.992746:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4895 00000100:00080000:0.0:1769077909.992748:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077909.992750:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077909.992753:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077909.992763:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616818048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077909.992775:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077909.992777:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077909.992779:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077909.992781:0:38:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4895 00000100:00080000:0.0:1769077909.992783:0:38:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077909.992786:0:38:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077909.992789:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077909.992796:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616818176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077909.992801:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077909.992808:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077909.992811:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077909.992813:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077909.992816:0:38:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4895 00000100:00080000:0.0:1769077909.992821:0:38:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077909.992823:0:38:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077909.992827:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077909.992834:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616818304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077909.992841:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077909/real 1769077909] req@ffff995b84643800 x1855007616817792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077909.992842:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077909.992849:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077909.992851:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077909.992858:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4895 00000100:00080000:0.0:1769077909.992862:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077909.992865:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077909.992869:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616817792/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077909.992870:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077909.992883:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616818432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077909.992884:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077909.992892:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769077909.992895:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077909.992899:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616817792:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077909.992900:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077909.992904:0:38:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077909.992909:0:38:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4895 00000100:00080000:0.0:1769077909.992912:0:38:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077909.992914:0:38:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077909.992917:0:38:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077909.992924:0:38:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616818560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077909.992927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077909.992959:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077909.992977:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077909/real 1769077909] req@ffff995b84641180 x1855007616817920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077909.993009:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616817920/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077909.993021:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077909.993025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077909.993029:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616817920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077909.993036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077909.993058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077909.993071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077909/real 1769077909] req@ffff995b84641500 x1855007616818048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077909.993087:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616818048/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077909.993095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077909.993097:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077909.993099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077909.993104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818176:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077909.993126:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077909.993134:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077909/real 1769077909] req@ffff995b84640000 x1855007616818176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077909.993142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616818176/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077909.993154:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077909.993157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077909.993159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818176:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077909.993163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818304:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077909.993174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077909.993180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077909/real 1769077909] req@ffff995b84640380 x1855007616818304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077909.993187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616818304/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077909.993198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077909.993200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077909.993207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818304:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077909.993211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616818432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077909.993225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077909.993231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077909/real 1769077909] req@ffff995b84641f80 x1855007616818432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077909.993239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616818432/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077909.993244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077909.993246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077909.993248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616818432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077909.993262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616818560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077909.993272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077909.993279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077909/real 1769077909] req@ffff995b84640a80 x1855007616818560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077909.993286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616818560/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077909.993291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077909.993292:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077909.993295:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616818560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077910.696540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077910.696594:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077911.016553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077911.016558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077911.016598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077911.016607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077911.720558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077911.720622:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077912.040579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077912.040579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077912.040618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077912.040619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077912.744556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077912.744632:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077913.064569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077913.064569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077913.064579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077913.064603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077913.768605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077913.768651:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077914.088547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077914.088549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077914.088565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077914.088589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077914.792564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077914.792632:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077915.112561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077915.112569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077915.112570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077915.112591:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077915.112600:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1769077915.112601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077915.112606:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077915.112619:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4900 00000100:00080000:3.0:1769077915.112631:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077915.112640:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077915.112682:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616818688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077915.112712:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077915.112716:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077915.112718:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077915.112722:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4900 00000100:00080000:3.0:1769077915.112726:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077915.112730:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077915.112740:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077915.112754:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616818816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077915.112757:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616818688:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077915.112765:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077915.112768:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077915.112770:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077915.112773:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4900 00000100:00080000:3.0:1769077915.112776:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077915.112779:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077915.112783:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077915.112794:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616818944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077915.112802:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077915.112805:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077915.112807:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077915.112810:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4900 00000100:00080000:3.0:1769077915.112821:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077915.112824:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077915.112829:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1769077915.112835:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077915.112838:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616819072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077915.112846:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077915.112849:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077915.112851:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077915.112854:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4900 00000100:00080000:3.0:1769077915.112857:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077915.112861:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077915.112865:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077915.112874:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616819200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077915.112875:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077915/real 1769077915] req@ffff995b84558000 x1855007616818688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077915.112884:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077915.112887:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077915.112888:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077915.112892:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4900 00000100:00100000:2.0:1769077915.112893:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616818688/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077915.112895:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077915.112900:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077915.112903:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769077915.112920:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077915.112932:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616819328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077915.112934:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077915.112938:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616818688:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077915.112941:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077915.112944:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077915.112946:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077915.112950:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4900 00000100:00080000:3.0:1769077915.112954:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077915.112957:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077915.112962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818816:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769077915.112962:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077915.112971:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616819456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077915.112985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077915.112998:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077915/real 1769077915] req@ffff995b84559500 x1855007616818816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077915.113012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616818816/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077915.113022:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077915.113025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077915.113027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077915.113032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077915.113048:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077915.113056:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077915/real 1769077915] req@ffff995b8455bb80 x1855007616818944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077915.113064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616818944/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077915.113071:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077915.113073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077915.113075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616818944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077915.113079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819072:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077915.113089:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077915.113095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077915/real 1769077915] req@ffff995b84558a80 x1855007616819072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077915.113102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616819072/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077915.113112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077915.113114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077915.113116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077915.113119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819200:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077915.113128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077915.113133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077915/real 1769077915] req@ffff995b8455a680 x1855007616819200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077915.113140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616819200/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077915.113146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077915.113148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077915.113150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819200:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077915.113153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616819328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077915.113162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077915.113167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077915/real 1769077915] req@ffff995b8455b100 x1855007616819328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077915.113173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616819328/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077915.113179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077915.113181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077915.113184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616819328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077915.113196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616819456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077915.113206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077915.113212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077915/real 1769077915] req@ffff995b8455aa00 x1855007616819456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077915.113219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616819456/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077915.113224:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077915.113225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077915.113227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616819456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077915.816616:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077915.816700:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077916.136600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077916.136607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077916.136612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077916.136635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077916.840564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077916.840627:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077917.160567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077917.160570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077917.160576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077917.160599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077917.864572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077917.864612:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077918.184569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077918.184569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077918.184668:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077918.184685:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077918.888604:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077918.888680:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077919.208593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077919.208593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077919.208593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077919.208595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077919.912618:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077919.912687:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077920.232529:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077920.232547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077920.232561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077920.232566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077920.232586:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077920.232593:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077920.232598:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077920.232610:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4905 00000100:00080000:2.0:1769077920.232624:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077920.232634:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077920.232675:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616819584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077920.232704:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077920.232708:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077920.232709:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077920.232713:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4905 00000100:00080000:2.0:1769077920.232717:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077920.232721:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077920.232731:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077920.232738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616819584:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077920.232743:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616819712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077920.232753:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077920.232755:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077920.232756:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077920.232759:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4905 00000100:00080000:2.0:1769077920.232762:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077920.232765:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077920.232768:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077920.232779:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616819840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077920.232786:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077920.232789:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077920.232791:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077920.232794:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4905 00000100:00080000:2.0:1769077920.232797:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077920.232799:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077920.232803:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077920.232810:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616819968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077920.232816:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077920.232836:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077920.232843:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077920.232846:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077920.232850:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4905 00000100:00080000:2.0:1769077920.232854:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077920.232858:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077920.232862:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077920.232877:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616820096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077920.232883:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077920/real 1769077920] req@ffff995b85ac8380 x1855007616819584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077920.232887:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077920.232892:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077920.232894:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077920.232902:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4905 00000100:00080000:2.0:1769077920.232907:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077920.232911:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077920.232914:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077920.232923:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616819584/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077920.232924:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616820224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077920.232951:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077920.232956:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077920.232958:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077920.232959:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077920.232971:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4905 00000100:00080000:1.0:1769077920.232973:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077920.232976:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077920.232981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616819584:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077920.232986:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077920.232994:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077920.233008:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616820352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077920.233016:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077920.233044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077920.233056:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077920/real 1769077920] req@ffff995b85aca300 x1855007616819712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077920.233065:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616819712/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077920.233072:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077920.233074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077920.233078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077920.233083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077920.233103:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077920.233115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077920/real 1769077920] req@ffff995b85acb100 x1855007616819840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077920.233128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616819840/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077920.233139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077920.233143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077920.233147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077920.233153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819968:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077920.233174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077920.233186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077920/real 1769077920] req@ffff995b85ac9c00 x1855007616819968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077920.233199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616819968/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077920.233208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077920.233212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077920.233216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616819968:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077920.233223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820096:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077920.233243:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077920.233255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077920/real 1769077920] req@ffff995b85acaa00 x1855007616820096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077920.233264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616820096/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077920.233270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077920.233272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077920.233275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820096:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077920.233279:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616820224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077920.233303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077920.233315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077920/real 1769077920] req@ffff995b85ac9880 x1855007616820224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077920.233323:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616820224/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077920.233330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077920.233332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077920.233334:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616820224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077920.233348:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616820352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077920.233360:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077920.233366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077920/real 1769077920] req@ffff995b85acbb80 x1855007616820352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077920.233373:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616820352/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077920.233380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077920.233382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077920.233384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616820352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077920.936564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077920.936633:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077921.256605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077921.256605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077921.256605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077921.256605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077921.960596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077921.960663:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077922.280537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077922.280542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077922.280586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077922.280640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077922.984572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077922.984637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077923.304567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077923.304567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077923.304620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077923.304730:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077924.008614:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077924.008693:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077924.328633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077924.328674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077924.328675:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077924.328717:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077925.032554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077925.032623:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077925.352580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077925.352589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077925.352596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077925.352635:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077925.352639:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769077925.352642:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077925.352648:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077925.352657:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4911 00000100:00080000:1.0:1769077925.352667:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077925.352674:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077925.352725:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616820480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077925.352768:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077925.352772:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077925.352773:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077925.352777:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4911 00000100:00080000:1.0:1769077925.352782:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077925.352786:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077925.352805:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077925.352807:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616820480:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077925.352822:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616820608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077925.352832:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077925.352835:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077925.352837:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077925.352840:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4911 00000100:00080000:1.0:1769077925.352843:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077925.352846:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077925.352850:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077925.352859:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616820736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077925.352867:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077925.352870:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077925.352871:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077925.352874:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4911 00000100:00080000:1.0:1769077925.352877:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077925.352879:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077925.352883:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077925.352887:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077925.352895:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616820864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077925.352904:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077925.352907:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077925.352908:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077925.352923:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4911 00000100:00080000:1.0:1769077925.352959:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077925.352963:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077925.352966:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077925.352976:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616820992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077925.352984:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077925/real 1769077925] req@ffff995b868c4a80 x1855007616820480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077925.352985:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077925.352991:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077925.352994:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077925.353001:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4911 00000100:00080000:1.0:1769077925.353005:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077925.353008:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077925.353011:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077925.353018:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616821120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077925.353024:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077925.353027:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077925.353028:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077925.353030:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4911 00000100:00080000:1.0:1769077925.353032:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077925.353035:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077925.353037:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077925.353043:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616821248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077925.353122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616820480/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077925.353133:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077925.353137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077925.353176:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616820480:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077925.353202:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077925.353223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077925.353233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077925/real 1769077925] req@ffff995b868c6d80 x1855007616820608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077925.353242:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616820608/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077925.353248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077925.353250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077925.353253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077925.353257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077925.353267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077925.353274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077925/real 1769077925] req@ffff995b868c6300 x1855007616820736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077925.353286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616820736/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077925.353293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077925.353295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077925.353304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077925.353311:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820864:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077925.353323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077925.353330:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077925/real 1769077925] req@ffff995b868c6a00 x1855007616820864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077925.353337:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616820864/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077925.353343:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077925.353345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077925.353347:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820864:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077925.353351:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820992:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077925.353360:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077925.353369:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077925/real 1769077925] req@ffff995b868c7100 x1855007616820992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077925.353381:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616820992/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077925.353397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077925.353402:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077925.353405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616820992:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077925.353411:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616821120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077925.353433:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077925.353446:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077925/real 1769077925] req@ffff995b868c7480 x1855007616821120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077925.353485:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616821120/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077925.353495:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077925.353499:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077925.353504:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616821120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077925.353513:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616821248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077925.353537:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077925.353549:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077925/real 1769077925] req@ffff995b868c7800 x1855007616821248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077925.353561:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616821248/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077925.353567:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077925.353569:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077925.353572:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616821248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077926.056576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077926.056633:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077926.376610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077926.376610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077926.376615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077926.376671:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077927.080543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077927.080636:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077927.400569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077927.400582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077927.400589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077927.400613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077928.104555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077928.104628:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077928.424565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077928.424565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077928.424571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077928.424594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077929.128568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077929.128641:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077929.448646:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077929.448650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077929.448652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077929.448680:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077930.152673:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077930.152685:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077930.472597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077930.472600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077930.472603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077930.472635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769077930.472677:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077930.472685:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077930.472690:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077930.472700:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4916 00000100:00080000:0.0:1769077930.472712:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077930.472719:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077930.472757:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616821376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077930.472781:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077930.472784:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077930.472785:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077930.472788:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4916 00000100:00080000:0.0:1769077930.472791:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077930.472794:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077930.472802:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077930.472810:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616821504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077930.472818:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077930.472821:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077930.472822:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077930.472825:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4916 00000100:00080000:0.0:1769077930.472827:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077930.472829:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077930.472832:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077930.472838:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616821376:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077930.472842:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616821632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077930.472852:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077930.472855:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077930.472856:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077930.472859:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4916 00000100:00080000:0.0:1769077930.472861:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077930.472863:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077930.472867:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077930.472876:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616821760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077930.472882:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077930.472886:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077930.472893:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077930.472896:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4916 00000100:00080000:0.0:1769077930.472899:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077930.472902:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077930.472905:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077930.472914:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616821888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077930.472921:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077930.472923:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077930.472924:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077930.472927:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4916 00000100:00080000:0.0:1769077930.472930:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077930.472932:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077930.472935:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077930.472943:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616822016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077930.472949:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077930.472953:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077930.472955:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077930.472959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616821504:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077930.472963:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4916 00000100:00080000:0.0:1769077930.472970:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077930.472975:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077930.472980:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077930.472988:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616822144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077930.473016:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616821632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077930.473047:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616821760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077930.473074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616821888:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077930.473093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616822016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077930.473115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616822144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077931.176570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077931.176648:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077931.496559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077931.496560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077931.496565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077931.496676:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077932.200557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077932.200647:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077932.520581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077932.520581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077932.520695:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077932.520764:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077933.224545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077933.224611:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077933.544577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077933.544590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077933.544747:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077933.544789:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:2.0:1769077933.544790:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769077933.544840:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077933.544862:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077933.544908:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077933.544919:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077933.544927:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077933.544934:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077933.544943:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769077933.544951:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769077933.545111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077930/real 1769077933] req@ffff995b84641c00 x1855007616821632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077933.545140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616821632/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077933.545175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077933.545181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077933.545186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616821632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077933.545195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077930/real 1769077933] req@ffff995b84640a80 x1855007616821760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077933.545209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616821760/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077933.545220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077933.545224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077933.545228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616821760:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077933.545236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077930/real 1769077933] req@ffff995b84641f80 x1855007616821888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077933.545249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616821888/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077933.545259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077933.545262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077933.545266:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616821888:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077933.545272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077930/real 1769077933] req@ffff995b84640380 x1855007616822016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077933.545290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616822016/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077933.545298:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077933.545302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077933.545305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616822016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077933.545312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077930/real 1769077933] req@ffff995b84640000 x1855007616822144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077933.545322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616822144/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077933.545332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077933.545335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077933.545339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616822144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077933.545346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077930/real 1769077933] req@ffff995b84642680 x1855007616821376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077933.545914:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616821376/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077933.545924:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077933.545928:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077933.545932:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616821376:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077933.545939:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077930/real 1769077933] req@ffff995b84640e00 x1855007616821504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077933.545950:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616821504/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077933.545959:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077933.545963:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077933.545966:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616821504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077934.248580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077934.248678:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077934.568613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077934.568617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077934.568644:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077934.568679:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077935.272609:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077935.272611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077935.592553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077935.592608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077935.592640:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077935.592646:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769077935.592650:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077935.592657:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077935.592657:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077935.592675:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4921 00000100:00080000:0.0:1769077935.592692:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077935.592703:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077935.592746:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616822272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077935.592785:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077935.592789:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077935.592791:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077935.592797:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4921 00000100:00080000:0.0:1769077935.592801:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077935.592806:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077935.592817:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077935.592840:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616822400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077935.592853:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077935.592861:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769077935.592861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616822272:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077935.592868:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077935.592882:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4921 00000100:00080000:0.0:1769077935.592886:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077935.592891:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077935.592897:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077935.592915:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616822528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077935.592927:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077935.592931:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077935.592934:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077935.592955:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4921 00000100:00080000:0.0:1769077935.592963:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077935.592968:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769077935.592975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769077935.592979:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077935.593005:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616822656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077935.593017:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077935.593022:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769077935.593032:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077935/real 1769077935] req@ffff995b84641500 x1855007616822272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077935.593042:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077935.593054:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4921 00000100:00080000:0.0:1769077935.593060:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077935.593067:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616822272/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077935.593069:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077935.593083:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769077935.593092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077935.593111:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616822784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077935.593111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077935.593127:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616822272:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077935.593156:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077935.593161:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077935.593164:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077935.593171:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4921 00000100:00080000:0.0:1769077935.593176:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077935.593183:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077935.593188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616822400:192.168.204.132@tcp:38:kworker.0 00000008:00100000:0.0:1769077935.593193:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077935.593224:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616822912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077935.593236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077935.593253:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077935.593258:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077935.593261:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077935.593270:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4921 00000100:00100000:3.0:1769077935.593270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077935/real 1769077935] req@ffff995b84641180 x1855007616822400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077935.593276:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077935.593289:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077935.593300:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077935.593321:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616823040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077935.593389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616822400/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077935.593403:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077935.593410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077935.593415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616822400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077935.593424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616822528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077935.593482:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077935.593497:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077935/real 1769077935] req@ffff995b84643800 x1855007616822528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077935.593511:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616822528/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077935.593522:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077935.593527:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077935.593531:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616822528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077935.593538:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616822656:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077935.593573:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077935.593587:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077935/real 1769077935] req@ffff995b84643480 x1855007616822656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077935.593599:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616822656/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077935.593608:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077935.593612:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077935.593615:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616822656:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077935.593623:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616822784:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077935.593644:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077935.593657:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077935/real 1769077935] req@ffff995b84642d80 x1855007616822784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077935.593668:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616822784/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077935.593678:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077935.593681:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077935.593684:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616822784:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077935.593707:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616822912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077935.593731:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077935.593744:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077935/real 1769077935] req@ffff995b84643100 x1855007616822912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077935.593756:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616822912/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077935.593765:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077935.593769:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077935.593773:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616822912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077935.593780:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616823040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077935.593801:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077935.593815:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077935/real 1769077935] req@ffff995b84643b80 x1855007616823040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077935.593836:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616823040/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077935.593845:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077935.593849:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077935.593852:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616823040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077936.296629:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077936.296707:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077936.616608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077936.616620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077936.616640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077936.616732:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077937.320604:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077937.320626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077937.640611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077937.640611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077937.640612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077937.640619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077938.344588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077938.344686:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077938.664603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077938.664603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077938.664603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077938.664648:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077939.368606:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077939.368677:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077939.688553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077939.688570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077939.688607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077939.688610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077940.392581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077940.392650:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077940.712610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077940.712610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769077940.712610:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077940.712628:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077940.712641:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077940.712654:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769077940.712654:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4926 00000100:00100000:2.0:1769077940.712669:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769077940.712671:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077940.712685:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077940.712735:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616823168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077940.712766:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077940.712774:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077940.712776:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077940.712781:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4926 00000100:00080000:3.0:1769077940.712786:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077940.712791:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077940.712800:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077940.712807:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616823168:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077940.712816:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616823296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077940.712825:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077940.712829:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077940.712831:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077940.712836:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4926 00000100:00080000:3.0:1769077940.712839:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077940.712843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077940.712846:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077940.712859:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616823424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077940.712865:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077940.712868:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077940.712869:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077940.712871:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4926 00000100:00080000:3.0:1769077940.712874:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077940.712876:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077940.712879:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077940.712886:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616823552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077940.712893:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077940.712897:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769077940.712904:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077940.712906:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077940.712911:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4926 00000100:00080000:3.0:1769077940.712916:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077940.712921:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077940.712926:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077940.712935:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616823680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077940.712943:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077940.712946:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077940.712948:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077940.712951:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4926 00000100:00080000:3.0:1769077940.712955:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077940.712956:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077940/real 1769077940] req@ffff995b8455aa00 x1855007616823168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077940.712962:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077940.712980:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077940.712988:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616823168/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077940.712992:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616823808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077940.713039:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077940.713040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077940.713056:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077940.713058:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077940.713066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616823168:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769077940.713066:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077940.713082:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4926 00000100:00080000:3.0:1769077940.713087:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077940.713090:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769077940.713095:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077940.713107:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616823936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077940.713111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616823296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077940.713150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077940.713167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077940/real 1769077940] req@ffff995b8455b100 x1855007616823296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077940.713175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616823296/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077940.713182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077940.713185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077940.713187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616823296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077940.713191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616823424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077940.713202:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077940.713208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077940/real 1769077940] req@ffff995b8455a680 x1855007616823424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077940.713215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616823424/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077940.713223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077940.713225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077940.713227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616823424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077940.713231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616823552:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077940.713245:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077940.713256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077940/real 1769077940] req@ffff995b84558a80 x1855007616823552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077940.713269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616823552/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077940.713278:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077940.713282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077940.713285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616823552:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077940.713293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616823680:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769077940.713313:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077940.713323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077940/real 1769077940] req@ffff995b8455bb80 x1855007616823680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077940.713331:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616823680/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077940.713338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077940.713340:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077940.713342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616823680:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769077940.713346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616823808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077940.713359:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077940.713365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077940/real 1769077940] req@ffff995b84559500 x1855007616823808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077940.713372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616823808/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077940.713382:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077940.713384:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077940.713386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616823808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077940.713399:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616823936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769077940.713410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077940.713417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077940/real 1769077940] req@ffff995b84558000 x1855007616823936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077940.713424:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616823936/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077940.713429:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077940.713431:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077940.713433:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616823936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077941.416596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077941.416672:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077941.736636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077941.736640:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077941.736684:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077941.736695:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077942.440555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077942.440646:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077942.760566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077942.760573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077942.760626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077942.760673:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077943.464618:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077943.464674:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077943.784559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077943.784565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077943.784585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077943.784652:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077944.488534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077944.488671:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077944.808529:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077944.808590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077944.808603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077944.808603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077945.512601:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077945.512661:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077945.832557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077945.832582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077945.832612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077945.832612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077945.832631:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077945.832638:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077945.832645:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077945.832657:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4931 00000100:00080000:1.0:1769077945.832671:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077945.832679:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077945.832731:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616824064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077945.832757:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077945.832759:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077945.832761:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077945.832764:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4931 00000100:00080000:1.0:1769077945.832767:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077945.832770:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077945.832778:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077945.832790:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616824064:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077945.832791:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616824192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077945.832805:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077945.832808:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077945.832809:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077945.832811:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4931 00000100:00080000:1.0:1769077945.832814:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077945.832816:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077945.832819:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077945.832827:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616824320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077945.832838:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077945.832841:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077945.832842:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077945.832844:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4931 00000100:00080000:1.0:1769077945.832854:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077945.832856:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769077945.832861:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769077945.832863:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077945.832875:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616824448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077945.832881:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077945.832884:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077945.832885:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077945.832888:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4931 00000100:00080000:1.0:1769077945.832891:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077945.832893:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077945.832896:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077945.832898:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077945/real 1769077945] req@ffff995b868c5c00 x1855007616824064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077945.832906:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616824576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077945.832914:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077945.832918:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616824064/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077945.832919:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077945.832923:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077945.832931:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4931 00000100:00080000:1.0:1769077945.832936:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077945.832936:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077945.832940:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077945.832945:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769077945.832947:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077945.832957:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616824704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077945.832961:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616824064:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077945.832965:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077945.832973:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077945.832975:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077945.832979:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4931 00000100:00080000:1.0:1769077945.832981:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077945.832984:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077945.832986:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077945.832994:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616824192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077945.832996:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616824832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077945.833022:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077945.833033:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077945/real 1769077945] req@ffff995b868c5500 x1855007616824192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077945.833042:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616824192/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077945.833054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077945.833056:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077945.833059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616824192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077945.833063:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616824320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077945.833082:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077945.833089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077945/real 1769077945] req@ffff995b868c4700 x1855007616824320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077945.833097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616824320/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077945.833104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077945.833107:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077945.833109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616824320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077945.833113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616824448:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077945.833125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077945.833130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077945/real 1769077945] req@ffff995b868c7800 x1855007616824448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077945.833138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616824448/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077945.833144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077945.833146:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077945.833149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616824448:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077945.833153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616824576:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077945.833170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077945.833180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077945/real 1769077945] req@ffff995b868c7480 x1855007616824576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077945.833193:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616824576/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077945.833202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077945.833206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077945.833209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616824576:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077945.833216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616824704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077945.833235:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077945.833244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077945/real 1769077945] req@ffff995b868c7100 x1855007616824704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077945.833253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616824704/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077945.833258:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077945.833261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077945.833264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616824704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077945.833282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616824832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077945.833304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077945.833317:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077945/real 1769077945] req@ffff995b868c6a00 x1855007616824832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077945.833327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616824832/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077945.833334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077945.833336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077945.833339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616824832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077946.536615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077946.536702:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077946.856578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077946.856578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077946.856578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077946.856678:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077947.560550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077947.560610:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077947.880522:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077947.880522:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077947.880532:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077947.880553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077948.584575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077948.584632:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077948.904527:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077948.904555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077948.904665:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077948.904680:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077949.608560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077949.608616:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077949.928573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077949.928573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077949.928618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077949.928643:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077950.632560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077950.632617:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077950.952552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077950.952568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077950.952571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077950.952607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077950.952614:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077950.952620:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077950.952624:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077950.952636:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4936 00000100:00080000:2.0:1769077950.952647:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077950.952653:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077950.952691:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616824960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077950.952713:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077950.952716:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077950.952717:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077950.952720:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4936 00000100:00080000:2.0:1769077950.952723:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077950.952725:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077950.952734:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077950.952743:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616825088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077950.952751:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077950.952754:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616824960:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077950.952755:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077950.952758:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077950.952766:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4936 00000100:00080000:2.0:1769077950.952768:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077950.952770:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077950.952773:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077950.952785:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616825216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077950.952792:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077950.952794:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077950.952796:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077950.952799:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4936 00000100:00080000:2.0:1769077950.952810:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077950.952813:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077950.952816:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077950.952822:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077950.952826:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616825344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077950.952834:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077950.952836:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077950.952838:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077950.952841:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4936 00000100:00080000:2.0:1769077950.952844:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077950.952846:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077950.952849:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077950.952858:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616825472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077950.952863:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077950/real 1769077950] req@ffff995b85ac8a80 x1855007616824960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077950.952869:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077950.952877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077950.952881:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077950.952886:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4936 00000100:00080000:2.0:1769077950.952889:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077950.952893:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616824960/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077950.952896:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077950.952903:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077950.952908:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077950.952913:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616825600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077950.952914:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077950.952922:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616824960:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077950.952929:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077950.952932:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077950.952933:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077950.952937:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4936 00000100:00080000:2.0:1769077950.952939:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077950.952941:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077950.952944:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077950.952951:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616825728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077950.952955:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077950.952987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077950.953009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077950/real 1769077950] req@ffff995b85acb800 x1855007616825088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077950.953019:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616825088/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077950.953025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077950.953027:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077950.953030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077950.953034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077950.953048:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077950.953055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077950/real 1769077950] req@ffff995b85acbb80 x1855007616825216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077950.953062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616825216/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077950.953069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077950.953071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077950.953073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077950.953077:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825344:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077950.953087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077950.953093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077950/real 1769077950] req@ffff995b85ac9880 x1855007616825344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077950.953101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616825344/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077950.953110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077950.953114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077950.953117:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825344:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077950.953123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825472:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077950.953144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077950.953155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077950/real 1769077950] req@ffff995b85acaa00 x1855007616825472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077950.953168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616825472/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077950.953175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077950.953177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077950.953179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825472:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077950.953183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616825600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077950.953197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077950.953205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077950/real 1769077950] req@ffff995b85ac9c00 x1855007616825600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077950.953212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616825600/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077950.953218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077950.953220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077950.953222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616825600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077950.953236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616825728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077950.953246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077950.953252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077950/real 1769077950] req@ffff995b85acb100 x1855007616825728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077950.953259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616825728/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077950.953264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077950.953266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077950.953268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616825728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077951.656557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077951.656637:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077951.976613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077951.976620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077951.976620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077951.976620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077952.680562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077952.680625:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077953.000618:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077953.000623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077953.000624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077953.000627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077953.704584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077953.704734:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077954.024619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077954.024621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077954.024622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077954.024706:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077954.728638:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077954.728712:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077955.048559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077955.048560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077955.048560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077955.048560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077955.752561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077955.752662:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077956.072611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077956.072612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077956.072612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077956.072617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077956.072689:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077956.072696:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077956.072701:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077956.072712:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4941 00000100:00080000:1.0:1769077956.072724:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077956.072731:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077956.072774:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616825856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077956.072803:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077956.072807:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077956.072809:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077956.072814:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4941 00000100:00080000:1.0:1769077956.072817:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077956.072821:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077956.072829:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077956.072842:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616825984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077956.072851:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077956.072853:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077956.072854:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077956.072861:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4941 00000100:00100000:3.0:1769077956.072863:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616825856:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077956.072864:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077956.072872:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077956.072879:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077956.072886:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616826112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077956.072892:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077956.072894:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077956.072895:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077956.072898:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4941 00000100:00080000:1.0:1769077956.072900:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077956.072903:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077956.072906:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077956.072914:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616826240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077956.072920:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077956.072922:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077956.072925:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077956.072928:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4941 00000100:00080000:1.0:1769077956.072930:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077956.072933:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077956.072938:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077956.072944:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616826368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077956.072952:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077956.072955:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077956.072957:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077956.072961:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4941 00000100:00080000:1.0:1769077956.072968:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077956.072968:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077956.072979:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077956.072987:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077956.073002:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616826496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077956.073012:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077956.073016:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077956.073019:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077956.073023:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4941 00000100:00080000:1.0:1769077956.073029:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077956.073035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077956/real 1769077956] req@ffff995b868c6300 x1855007616825856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077956.073036:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077956.073046:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077956.073058:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616826624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077956.073115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616825856/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077956.073131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077956.073137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077956.073143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616825856:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769077956.073173:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077956.073205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077956.073251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077956/real 1769077956] req@ffff995b868c6d80 x1855007616825984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077956.073266:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616825984/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077956.073276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077956.073280:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077956.073284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616825984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077956.073292:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616826112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077956.073316:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077956.073328:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077956/real 1769077956] req@ffff995b868c4a80 x1855007616826112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077956.073341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616826112/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077956.073353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077956.073357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077956.073361:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616826112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077956.073368:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616826240:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077956.073409:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077956.073423:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077956/real 1769077956] req@ffff995b868c6a00 x1855007616826240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077956.073438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616826240/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077956.073449:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077956.073480:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077956.073486:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616826240:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077956.073494:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616826368:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077956.073518:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077956.073532:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077956/real 1769077956] req@ffff995b868c7100 x1855007616826368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077956.073545:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616826368/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077956.073556:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077956.073560:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077956.073564:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616826368:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077956.073571:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616826496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077956.073592:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077956.073605:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077956/real 1769077956] req@ffff995b868c7480 x1855007616826496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077956.073618:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616826496/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077956.073628:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077956.073632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077956.073635:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616826496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077956.073642:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616826624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077956.073666:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077956.073678:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077956/real 1769077956] req@ffff995b868c7800 x1855007616826624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077956.073690:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616826624/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077956.073699:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077956.073702:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077956.073705:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616826624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077956.776565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077956.776690:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077957.096537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077957.096542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077957.096564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077957.096584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077957.800557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077957.800611:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077958.120578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077958.120578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077958.120578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077958.120597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077958.824569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077958.824635:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077959.144531:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077959.144532:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077959.144536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077959.144557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077959.848544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077959.848649:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077960.168514:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077960.168537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077960.168553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077960.168573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077960.872589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077960.872692:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077961.192553:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077961.192568:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077961.192575:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077961.192588:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4946 00000100:00080000:1.0:1769077961.192602:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077961.192614:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077961.192616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077961.192619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077961.192654:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616826752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077961.192656:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077961.192692:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077961.192697:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077961.192699:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077961.192706:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4946 00000100:00080000:1.0:1769077961.192711:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077961.192717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616826752:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077961.192722:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077961.192738:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077961.192739:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077961.192761:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616826880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077961.192773:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077961.192777:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077961.192779:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077961.192783:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4946 00000100:00080000:1.0:1769077961.192787:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077961.192791:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077961.192797:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077961.192810:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616827008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077961.192812:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077961.192826:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077961.192829:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077961.192832:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077961.192836:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4946 00000100:00080000:1.0:1769077961.192900:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077961.192905:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077961.192911:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077961.192922:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616827136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077961.192930:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077961.192933:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077961.192935:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077961.192937:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4946 00000100:00080000:1.0:1769077961.192940:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077961.192943:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077961.192946:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077961.192956:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077961/real 1769077961] req@ffff995b868c4700 x1855007616826752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077961.192968:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616827264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077961.193046:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077961.193050:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077961.193052:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077961.193062:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4946 00000100:00100000:3.0:1769077961.193063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616826752/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077961.193076:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077961.193082:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077961.193087:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077961.193091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077961.193099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616826752:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769077961.193101:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077961.193121:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616827392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077961.193128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616826880:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077961.193153:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077961.193157:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077961.193159:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077961.193164:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4946 00000100:00080000:1.0:1769077961.193170:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077961.193175:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077961.193181:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077961.193184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077961.193199:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616827520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077961.193204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077961/real 1769077961] req@ffff995b868c5500 x1855007616826880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077961.193254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616826880/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077961.193261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077961.193265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077961.193270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616826880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077961.193276:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077961.193290:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077961.193296:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077961/real 1769077961] req@ffff995b868c5c00 x1855007616827008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077961.193304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616827008/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077961.193315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077961.193318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077961.193320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077961.193324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827136:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077961.193335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077961.193341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077961/real 1769077961] req@ffff995b868c7800 x1855007616827136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077961.193347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616827136/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077961.193530:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077961.193537:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077961.193542:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827136:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077961.193598:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827264:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077961.193631:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077961.193645:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077961/real 1769077961] req@ffff995b868c7480 x1855007616827264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077961.193656:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616827264/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077961.193663:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077961.193666:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077961.193669:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077961.193683:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616827392:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077961.193699:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077961.193710:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077961/real 1769077961] req@ffff995b868c7100 x1855007616827392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077961.193723:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616827392/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077961.193729:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077961.193732:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077961.193734:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616827392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077961.193739:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616827520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077961.193753:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077961.193760:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077961/real 1769077961] req@ffff995b868c6a00 x1855007616827520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077961.193768:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616827520/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077961.193773:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077961.193775:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077961.193777:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616827520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077961.896540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077961.896600:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077962.216549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077962.216581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077962.216588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077962.216605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077962.920554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077962.920593:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077963.240646:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077963.240646:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077963.240657:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077963.240729:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077963.944506:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077963.944624:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077964.264579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077964.264600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077964.264604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077964.264605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077964.968535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077964.968591:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077965.288535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077965.288555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077965.288583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077965.288628:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077965.992566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077965.992617:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077966.312542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077966.312551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077966.312574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077966.312604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769077966.312673:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077966.312685:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077966.312690:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077966.312703:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4952 00000100:00080000:1.0:1769077966.312717:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077966.312725:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077966.312768:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616827648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077966.312814:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077966.312819:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077966.312822:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077966.312828:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4952 00000100:00080000:1.0:1769077966.312833:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077966.312839:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077966.312851:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077966.312858:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616827648:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077966.312892:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616827776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077966.312909:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077966.312914:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077966.312916:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077966.312921:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4952 00000100:00080000:1.0:1769077966.312926:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077966.312930:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077966.312937:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077966.312942:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077966.312954:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616827904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077966.312967:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077966.312972:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077966.312975:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077966.312981:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077966/real 1769077966] req@ffff995b868c4a80 x1855007616827648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077966.312982:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4952 00000100:00080000:1.0:1769077966.312993:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077966.313006:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077966.313013:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616827648/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077966.313015:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769077966.313033:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077966.313038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077966.313042:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616827648:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769077966.313077:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616828032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077966.313092:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077966.313097:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077966.313100:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077966.313105:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827776:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077966.313110:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4952 00000100:00080000:1.0:1769077966.313118:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077966.313124:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077966.313130:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077966.313137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077966.313148:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616828160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077966.313153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077966/real 1769077966] req@ffff995b868c6d80 x1855007616827776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077966.313218:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077966.313226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616827776/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077966.313228:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077966.313234:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077966.313255:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4952 00000100:00080000:3.0:1769077966.313257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077966.313268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769077966.313270:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077966.313273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827776:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077966.313281:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077966.313284:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827904:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769077966.313294:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077966.313306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769077966.313317:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616828288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077966.313319:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077966/real 1769077966] req@ffff995b868c6300 x1855007616827904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077966.313386:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077966.313393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616827904/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077966.313394:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077966.313399:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077966.313411:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4952 00000100:00080000:1.0:1769077966.313422:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077966.313429:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769077966.313432:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769077966.313436:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077966.313442:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077966.313451:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616827904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769077966.313485:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616828416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077966.313490:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828032:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077966.313543:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077966.313559:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077966/real 1769077966] req@ffff995b868c6a00 x1855007616828032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077966.313574:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616828032/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077966.313584:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077966.313588:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077966.313591:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828032:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077966.313610:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077966.313627:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077966.313635:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077966/real 1769077966] req@ffff995b868c7100 x1855007616828160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077966.313648:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616828160/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077966.313655:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077966.313657:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077966.313660:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077966.313666:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616828288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077966.313678:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077966.313685:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077966/real 1769077966] req@ffff995b868c7480 x1855007616828288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077966.313692:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616828288/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077966.313698:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077966.313700:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077966.313701:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616828288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077966.313705:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616828416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077966.313715:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077966.313720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077966/real 1769077966] req@ffff995b868c7800 x1855007616828416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077966.313727:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616828416/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077966.313737:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077966.313739:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077966.313741:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616828416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077967.016550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077967.016672:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077967.336568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077967.336588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077967.336588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077967.336626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077968.040570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077968.040647:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077968.360567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077968.360567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077968.360578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077968.360619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077969.064562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077969.064633:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077969.384570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077969.384571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077969.384574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077969.384589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077970.088575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077970.088634:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077970.408570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077970.408570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077970.408570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077970.408577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077971.112573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077971.112650:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077971.432557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077971.432557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077971.432570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077971.432576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077971.432595:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077971.432601:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077971.432605:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077971.432613:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4957 00000100:00080000:2.0:1769077971.432625:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077971.432631:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077971.432668:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616828544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077971.432688:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077971.432691:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077971.432693:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077971.432696:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4957 00000100:00080000:2.0:1769077971.432699:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077971.432702:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077971.432711:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077971.432720:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616828672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077971.432727:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616828544:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077971.432732:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077971.432737:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077971.432738:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077971.432743:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4957 00000100:00080000:2.0:1769077971.432747:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077971.432752:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077971.432757:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077971.432774:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616828800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077971.432785:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077971.432789:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077971.432791:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077971.432796:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077971.432801:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4957 00000100:00080000:2.0:1769077971.432807:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077971.432812:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077971.432817:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077971.432830:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616828928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077971.432838:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077971/real 1769077971] req@ffff995b85aca300 x1855007616828544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077971.432839:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077971.432848:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077971.432851:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077971.432859:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4957 00000100:00080000:2.0:1769077971.432863:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077971.432866:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077971.432871:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769077971.432871:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616828544/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077971.432884:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077971.432887:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616829056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077971.432892:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077971.432900:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616828544:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077971.432904:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077971.432908:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077971.432909:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077971.432913:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4957 00000100:00080000:2.0:1769077971.432916:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077971.432918:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077971.432921:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077971.432926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077971.432932:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616829184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077971.432940:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077971.432944:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077971.432946:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077971.432950:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077971.432954:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4957 00000100:00080000:2.0:1769077971.432985:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077971.432991:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769077971.432991:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077971/real 1769077971] req@ffff995b85ac8380 x1855007616828672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077971.433000:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077971.433009:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616828672/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077971.433015:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616829312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077971.433025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077971.433033:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077971.433038:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077971.433044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077971.433059:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077971.433066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077971/real 1769077971] req@ffff995b85acb100 x1855007616828800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077971.433074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616828800/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077971.433080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077971.433082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077971.433084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077971.433088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828928:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077971.433098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077971.433105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077971/real 1769077971] req@ffff995b85ac9c00 x1855007616828928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077971.433116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616828928/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077971.433125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077971.433128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077971.433132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616828928:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077971.433138:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829056:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077971.433159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077971.433171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077971/real 1769077971] req@ffff995b85acaa00 x1855007616829056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077971.433179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616829056/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077971.433185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077971.433188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077971.433190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077971.433205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616829184:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077971.433225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077971.433237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077971/real 1769077971] req@ffff995b85ac9880 x1855007616829184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077971.433251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616829184/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077971.433260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077971.433263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077971.433265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616829184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077971.433269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616829312:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077971.433284:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077971.433292:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077971/real 1769077971] req@ffff995b85acbb80 x1855007616829312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077971.433299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616829312/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077971.433305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077971.433308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077971.433310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616829312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077972.136571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077972.136646:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077972.456556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077972.456556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077972.456568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077972.456601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077973.160537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077973.160593:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077973.480575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077973.480582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077973.480583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077973.480619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077974.184600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077974.184678:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077974.504538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077974.504541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077974.504557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077974.504592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077975.208530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077975.208591:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077975.528569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077975.528575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077975.528577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077975.528607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077976.232579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077976.232662:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077976.552539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077976.552542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077976.552546:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077976.552546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077976.552554:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077976.552559:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077976.552568:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4962 00000100:00080000:2.0:1769077976.552578:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077976.552583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077976.552587:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769077976.552626:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616829440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077976.552649:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077976.552652:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077976.552654:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077976.552657:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4962 00000100:00080000:2.0:1769077976.552660:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077976.552665:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077976.552673:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077976.552681:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616829440:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077976.552685:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616829568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077976.552695:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077976.552697:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077976.552699:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077976.552701:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4962 00000100:00080000:2.0:1769077976.552704:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077976.552706:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077976.552709:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077976.552720:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616829696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077976.552726:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077976.552729:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077976.552730:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077976.552732:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4962 00000100:00080000:2.0:1769077976.552736:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077976.552738:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077976.552741:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769077976.552748:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077976.552752:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616829824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077976.552759:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077976.552762:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077976.552764:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077976.552767:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4962 00000100:00080000:2.0:1769077976.552770:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077976.552773:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077976.552776:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077976.552783:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616829952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077976.552788:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077976.552791:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077976.552792:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077976.552797:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4962 00000100:00100000:3.0:1769077976.552799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077976/real 1769077976] req@ffff995b85acb800 x1855007616829440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077976.552800:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077976.552805:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077976.552811:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077976.552822:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616830080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077976.552828:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077976.552832:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769077976.552833:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616829440/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077976.552834:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077976.552843:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4962 00000100:00080000:2.0:1769077976.552849:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077976.552851:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077976.552852:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077976.552859:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769077976.552859:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077976.552864:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616829440:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769077976.552870:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616830208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077976.552912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077976.552953:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077976.552967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077976/real 1769077976] req@ffff995b85ac8a80 x1855007616829568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077976.552977:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616829568/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077976.552983:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077976.552986:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077976.552988:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077976.552993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077976.553007:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077976.553013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077976/real 1769077976] req@ffff995b85acbb80 x1855007616829696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077976.553028:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616829696/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077976.553036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077976.553038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077976.553040:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077976.553045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829824:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077976.553056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077976.553062:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077976/real 1769077976] req@ffff995b85ac9880 x1855007616829824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077976.553075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616829824/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077976.553081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077976.553083:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077976.553085:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829824:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077976.553089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829952:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769077976.553098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077976.553103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077976/real 1769077976] req@ffff995b85acaa00 x1855007616829952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077976.553111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616829952/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077976.553117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077976.553119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077976.553120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616829952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769077976.553124:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616830080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077976.553133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077976.553139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077976/real 1769077976] req@ffff995b85ac9c00 x1855007616830080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077976.553146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616830080/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077976.553156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077976.553158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077976.553159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616830080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077976.553163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616830208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769077976.553178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769077976.553185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077976/real 1769077976] req@ffff995b85acb100 x1855007616830208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077976.553192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616830208/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077976.553198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077976.553200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769077976.553202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616830208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077977.256560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077977.256610:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077977.576613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077977.576627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077977.576629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769077977.576676:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077978.280563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077978.280643:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077978.600548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077978.600555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077978.600605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077978.600620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077979.304576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077979.304807:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077979.624586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077979.624586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077979.624620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077979.624655:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077980.328569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077980.328630:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077980.648517:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077980.648531:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077980.648531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077980.648576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077981.352550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077981.352623:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077981.672560:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077981.672573:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769077981.672575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769077981.672579:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077981.672590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077981.672592:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4967 00000100:00080000:0.0:1769077981.672607:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077981.672614:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077981.672651:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616830336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077981.672673:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077981.672676:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077981.672678:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077981.672681:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4967 00000100:00080000:0.0:1769077981.672685:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077981.672689:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077981.672697:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077981.672705:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616830464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077981.672713:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077981.672716:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077981.672718:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077981.672721:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4967 00000100:00080000:0.0:1769077981.672724:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077981.672728:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077981.672733:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077981.672739:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616830336:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769077981.672749:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616830592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077981.672761:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077981.672764:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077981.672766:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077981.672770:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4967 00000100:00080000:0.0:1769077981.672774:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077981.672778:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077981.672783:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077981.672835:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616830720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077981.672849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769077981.672858:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077981.672864:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077981.672867:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077981.672873:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4967 00000100:00080000:0.0:1769077981.672881:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077981.672886:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077981.672892:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077981.672903:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077981/real 1769077981] req@ffff995b84641880 x1855007616830336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077981.672908:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616830848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077981.672932:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077981.672935:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077981.672939:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077981.672939:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616830336/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077981.672947:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4967 00000100:00080000:0.0:1769077981.672954:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077981.672960:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769077981.672962:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769077981.672964:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077981.672971:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077981.672977:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616830976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077981.672979:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616830336:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769077981.672994:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077981.672997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616830464:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769077981.672998:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077981.673002:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077981.673009:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4967 00000100:00080000:0.0:1769077981.673014:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077981.673018:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769077981.673029:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077981.673040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077981.673046:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616831104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077981.673058:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077981/real 1769077981] req@ffff995b84640700 x1855007616830464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077981.673075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616830464/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077981.673086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077981.673090:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077981.673091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077981.673098:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616830464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077981.673127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616830592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077981.673147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077981.673158:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077981/real 1769077981] req@ffff995b84642a00 x1855007616830592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077981.673172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616830592/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077981.673183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077981.673187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077981.673191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616830592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077981.673198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616830720:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077981.673215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077981.673234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077981/real 1769077981] req@ffff995b84643b80 x1855007616830720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077981.673247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616830720/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077981.673263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077981.673267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077981.673271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616830720:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077981.673277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616830848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077981.673302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077981.673319:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077981/real 1769077981] req@ffff995b84643100 x1855007616830848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077981.673332:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616830848/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077981.673342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077981.673345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077981.673349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616830848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077981.673355:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616830976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077981.673374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077981.673385:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077981/real 1769077981] req@ffff995b84642d80 x1855007616830976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077981.673399:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616830976/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077981.673407:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077981.673410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077981.673414:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616830976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077981.673420:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616831104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077981.673439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077981.673449:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077981/real 1769077981] req@ffff995b84643480 x1855007616831104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077981.673490:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616831104/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077981.673500:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077981.673504:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077981.673507:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616831104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769077981.673576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077982.376568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077982.376637:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077982.696555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077982.696555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077982.696555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077982.696577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077983.400528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077983.400620:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077983.720593:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077983.720593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077983.720609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077983.720610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077984.424565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077984.424648:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077984.744563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077984.744563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077984.744678:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077984.744836:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077985.448541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077985.448596:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077985.768556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077985.768556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077985.768556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077985.768587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077986.472586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077986.472685:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077986.792600:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077986.792603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077986.792650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077986.792659:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769077986.792668:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077986.792677:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077986.792689:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077986.792714:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4972 00000100:00080000:1.0:1769077986.792726:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077986.792735:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077986.792778:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616831232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077986.792806:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077986.792810:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077986.792812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077986.792816:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4972 00000100:00080000:1.0:1769077986.792820:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077986.792825:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077986.792835:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077986.792852:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616831360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077986.792852:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616831232:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077986.792869:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077986.792872:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077986.792874:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077986.792880:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4972 00000100:00080000:1.0:1769077986.792884:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077986.792887:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077986.792892:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077986.792905:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616831488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077986.792915:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077986.792919:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077986.792921:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077986.792925:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4972 00000100:00080000:1.0:1769077986.792928:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077986.792933:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769077986.792936:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077986.792943:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769077986.792957:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616831616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077986.792967:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077986.792971:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077986.792972:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077986.792976:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4972 00000100:00080000:1.0:1769077986.792980:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077986.792984:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077986.792989:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077986.792992:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077986/real 1769077986] req@ffff995b868c5c00 x1855007616831232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077986.793004:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616831744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077986.793014:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077986.793017:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077986.793022:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077986.793022:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616831232/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077986.793028:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4972 00000100:00080000:1.0:1769077986.793035:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077986.793041:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077986.793048:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769077986.793054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077986.793061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769077986.793064:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616831872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077986.793069:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616831232:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769077986.793082:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077986.793086:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077986.793089:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077986.793093:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4972 00000100:00080000:1.0:1769077986.793097:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077986.793102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616831360:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769077986.793103:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769077986.793111:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077986.793126:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616832000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077986.793137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077986.793155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077986/real 1769077986] req@ffff995b868c5500 x1855007616831360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077986.793169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616831360/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077986.793177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077986.793181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077986.793183:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616831360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077986.793188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616831488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077986.793203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077986.793210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077986/real 1769077986] req@ffff995b868c4700 x1855007616831488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077986.793217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616831488/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077986.793224:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077986.793226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077986.793228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616831488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077986.793232:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616831616:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077986.793242:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077986.793248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077986/real 1769077986] req@ffff995b868c7800 x1855007616831616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077986.793255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616831616/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077986.793261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077986.793263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077986.793264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616831616:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077986.793267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616831744:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769077986.793276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077986.793288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077986/real 1769077986] req@ffff995b868c7480 x1855007616831744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077986.793295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616831744/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077986.793301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077986.793303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077986.793305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616831744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077986.793309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616831872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077986.793319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077986.793325:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077986/real 1769077986] req@ffff995b868c7100 x1855007616831872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077986.793341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616831872/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077986.793349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077986.793352:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077986.793355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616831872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077986.793371:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616832000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769077986.793391:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769077986.793401:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077986/real 1769077986] req@ffff995b868c6a00 x1855007616832000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077986.793419:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616832000/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077986.793426:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077986.793429:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769077986.793432:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616832000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077987.496600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077987.496731:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077987.816602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077987.816622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077987.816674:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077987.817052:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077988.520571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077988.520632:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077988.840612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077988.840612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077988.840616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077988.840660:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077989.544554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077989.544618:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077989.864564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077989.864564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077989.864682:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077989.864694:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077990.568561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077990.568690:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077990.888554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077990.888556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077990.888735:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769077990.888820:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077991.592558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077991.592605:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077991.912574:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077991.912576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769077991.912585:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077991.912590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077991.912603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077991.912603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077991.912603:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4977 00000100:00080000:2.0:1769077991.912634:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077991.912644:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077991.912656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077991.912688:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616832128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077991.912714:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077991.912718:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077991.912720:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077991.912724:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4977 00000100:00080000:2.0:1769077991.912728:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077991.912734:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077991.912735:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616832128:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077991.912752:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077991.912768:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616832256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077991.912779:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077991.912783:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077991.912785:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077991.912789:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4977 00000100:00080000:2.0:1769077991.912793:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077991.912797:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769077991.912801:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077991.912805:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077991.912817:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616832384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077991.912824:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077991.912827:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077991.912828:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077991.912832:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077991/real 1769077991] req@ffff995b85ac8380 x1855007616832128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077991.912833:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4977 00000100:00080000:2.0:1769077991.912838:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077991.912843:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077991.912848:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769077991.912854:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616832128/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077991.912858:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616832512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077991.912873:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077991.912874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077991.912876:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077991.912879:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077991.912884:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769077991.912884:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4977 00000100:00080000:2.0:1769077991.912886:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077991.912891:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077991.912895:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616832128:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769077991.912895:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077991.912905:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616832640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077991.912911:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077991.912913:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077991.912914:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077991.912917:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4977 00000100:00080000:2.0:1769077991.912920:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077991.912924:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769077991.912926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616832256:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077991.912928:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077991.912937:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616832768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077991.912943:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077991.912945:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077991.912947:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077991.912949:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4977 00000100:00080000:2.0:1769077991.912951:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077991.912955:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769077991.912957:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769077991.912959:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077991.912968:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616832896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077991.912976:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077991/real 1769077991] req@ffff995b85aca300 x1855007616832256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077991.912994:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616832256/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077991.913003:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077991.913007:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077991.913010:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616832256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077991.913017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616832384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077991.913038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077991.913049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077991/real 1769077991] req@ffff995b85acb100 x1855007616832384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077991.913073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616832384/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077991.913089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077991.913093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077991.913097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616832384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077991.913104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616832512:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077991.913124:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077991.913135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077991/real 1769077991] req@ffff995b85ac9c00 x1855007616832512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077991.913145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616832512/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077991.913153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077991.913156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077991.913159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616832512:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077991.913164:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616832640:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769077991.913181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077991.913191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077991/real 1769077991] req@ffff995b85acaa00 x1855007616832640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077991.913202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616832640/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077991.913211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077991.913214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077991.913217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616832640:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769077991.913236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616832768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077991.913255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077991.913265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077991/real 1769077991] req@ffff995b85ac9880 x1855007616832768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077991.913275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616832768/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077991.913292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077991.913295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077991.913298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616832768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769077991.913303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616832896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769077991.913318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769077991.913327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077991/real 1769077991] req@ffff995b85acbb80 x1855007616832896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077991.913337:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616832896/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077991.913345:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077991.913348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769077991.913351:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616832896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077992.616561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077992.616627:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077992.936571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077992.936582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769077992.936583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077992.936622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077993.640565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077993.640647:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077993.960566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077993.960566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769077993.960566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077993.960621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077994.664562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077994.664670:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077994.984601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077994.984602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769077994.984603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077994.984651:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077995.688581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077995.688642:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077996.008537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769077996.008537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077996.008561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769077996.008583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769077996.712559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077996.712619:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077997.032613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769077997.032613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769077997.032632:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077997.032633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769077997.032645:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077997.032650:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077997.032661:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4982 00000100:00080000:2.0:1769077997.032673:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077997.032681:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077997.032688:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769077997.032722:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616833024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077997.032761:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077997.032768:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077997.032771:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077997.032776:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4982 00000100:00080000:2.0:1769077997.032781:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077997.032782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616833024:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769077997.032788:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077997.032798:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077997.032816:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616833152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077997.032831:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077997.032835:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077997.032838:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077997.032842:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4982 00000100:00080000:2.0:1769077997.032846:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077997.032850:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077997.032856:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077997.032874:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616833280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077997.032885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616833152:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077997.032889:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077997.032894:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077997.032896:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077997.032902:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4982 00000100:00080000:2.0:1769077997.032907:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077997.032912:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769077997.032915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616833280:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769077997.032920:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077997.032937:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616833408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077997.032949:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077997.032953:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077997.032955:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077997.032960:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4982 00000100:00080000:2.0:1769077997.032965:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077997.032970:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077997.032976:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769077997.032993:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616833536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077997.032999:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616833408:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769077997.033013:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077997.033018:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077997.033022:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077997.033030:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4982 00000100:00080000:2.0:1769077997.033036:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077997.033043:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077997.033051:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077997.033070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616833536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769077997.033071:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616833664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077997.033087:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077997.033092:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077997.033095:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077997.033103:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4982 00000100:00080000:2.0:1769077997.033109:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077997.033114:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616833664:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769077997.033118:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769077997.033127:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077997.033144:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616833792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077997.033171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616833792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769077997.736580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077997.736640:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077998.056550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769077998.056553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077998.056585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769077998.056626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077998.760548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077998.760603:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077999.080572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769077999.080576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769077999.080601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769077999.080614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769077999.784573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077999.784663:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078000.104580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078000.104600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078000.104611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078000.104635:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:2.0:1769078000.104788:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769078000.104835:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078000.104857:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078000.104921:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078000.104932:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078000.104940:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078000.104948:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078000.104956:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078000.104965:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:1.0:1769078000.105006:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077997/real 1769078000] req@ffff995b85acb100 x1855007616833792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078000.105040:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616833792/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078000.105053:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078000.105059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078000.105063:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616833792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078000.105071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077997/real 1769078000] req@ffff995b85ac9c00 x1855007616833664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078000.105081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616833664/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078000.105087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078000.105091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078000.105093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616833664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078000.105099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077997/real 1769078000] req@ffff995b85acaa00 x1855007616833536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078000.105110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616833536/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078000.105118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078000.105122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078000.105125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616833536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078000.105132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077997/real 1769078000] req@ffff995b85ac9880 x1855007616833408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078000.105142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616833408/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078000.105151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078000.105154:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078000.105157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616833408:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078000.105164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077997/real 1769078000] req@ffff995b85acbb80 x1855007616833280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078000.105191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616833280/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078000.105201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078000.105204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078000.105207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616833280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078000.105213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077997/real 1769078000] req@ffff995b85acb800 x1855007616833152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078000.105224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616833152/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078000.105231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078000.105235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078000.105238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616833152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078000.105245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077997/real 1769078000] req@ffff995b85ac8a80 x1855007616833024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078000.105259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616833024/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078000.105265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078000.105268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078000.105272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616833024:192.168.204.132@tcp:250:kworker.0 00010000:00010000:1.0:1769078000.808581:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078000.808603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078001.128573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078001.128573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078001.128621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078001.128622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078001.832552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078001.832608:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078002.152532:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078002.152537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078002.152561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078002.152583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078002.152588:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078002.152595:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078002.152600:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078002.152609:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4987 00000100:00080000:2.0:1769078002.152619:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078002.152626:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078002.152664:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616833920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078002.152692:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078002.152696:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078002.152698:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078002.152702:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4987 00000100:00080000:2.0:1769078002.152705:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078002.152709:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078002.152718:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078002.152732:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616834048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078002.152740:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078002.152743:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078002.152745:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078002.152748:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4987 00000100:00080000:2.0:1769078002.152751:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078002.152754:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078002.152759:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078002.152769:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616834176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078002.152778:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078002.152781:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078002.152783:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078002.152787:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4987 00000100:00080000:2.0:1769078002.152790:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078002.152794:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078002.152800:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078002.152801:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616833920:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078002.152812:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616834304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078002.152822:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078002.152825:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078002.152827:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078002.152830:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4987 00000100:00080000:2.0:1769078002.152832:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078002.152836:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078002.152840:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078002.152850:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616834432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078002.152858:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078002.152861:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078002.152864:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078002.152867:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4987 00000100:00080000:2.0:1769078002.152872:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078002.152876:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078002.152880:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078002.152887:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078002.152899:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616834560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078002.152907:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078002.152911:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078002.152913:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078002.152917:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4987 00000100:00080000:2.0:1769078002.152921:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078002.152925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078002/real 1769078002] req@ffff995b85aca300 x1855007616833920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078002.152928:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078002.152935:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078002.152946:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616834688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078002.153201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616833920/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078002.153217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078002.153224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078002.153229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616833920:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078002.153244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078002.153312:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078002.153340:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078002/real 1769078002] req@ffff995b85ac8380 x1855007616834048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078002.153378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616834048/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078002.153388:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078002.153392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078002.153396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078002.153403:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078002.153428:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078002.153537:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078002/real 1769078002] req@ffff995b85ac8a80 x1855007616834176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078002.153553:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616834176/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078002.153565:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078002.153570:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078002.153573:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078002.153617:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834304:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078002.153646:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078002.153658:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078002/real 1769078002] req@ffff995b85acb800 x1855007616834304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078002.153672:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616834304/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078002.153692:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078002.153696:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078002.153700:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834304:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078002.153709:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834432:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078002.153733:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078002.153761:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078002/real 1769078002] req@ffff995b85acbb80 x1855007616834432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078002.153776:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616834432/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078002.153787:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078002.153792:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078002.153796:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834432:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078002.153805:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616834560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078002.153831:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078002.153844:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078002/real 1769078002] req@ffff995b85ac9880 x1855007616834560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078002.153858:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616834560/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078002.153875:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078002.153878:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078002.153883:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616834560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078002.153891:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616834688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078002.153914:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078002.153927:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078002/real 1769078002] req@ffff995b85acaa00 x1855007616834688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078002.153941:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616834688/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078002.153951:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078002.153955:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078002.153958:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616834688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078002.856548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078002.856599:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078003.176583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078003.176590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078003.176623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078003.176674:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078003.880567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078003.880634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078004.200562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078004.200562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078004.200600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078004.200628:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078004.904528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078004.904569:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078005.224552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078005.224552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078005.224553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078005.224569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078005.928543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078005.928584:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078006.248542:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078006.248542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078006.248542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078006.248543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078006.952550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078006.952605:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078007.272538:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078007.272538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078007.272557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078007.272560:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078007.272566:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078007.272570:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078007.272579:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4992 00000100:00080000:3.0:1769078007.272594:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078007.272595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078007.272601:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078007.272635:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616834816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078007.272657:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078007.272659:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078007.272660:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078007.272663:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4992 00000100:00080000:3.0:1769078007.272665:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078007.272668:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078007.272676:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078007.272683:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616834816:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078007.272686:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616834944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078007.272693:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078007.272695:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078007.272696:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078007.272699:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4992 00000100:00080000:3.0:1769078007.272701:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078007.272703:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078007.272705:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078007.272711:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616835072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078007.272717:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078007.272719:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078007.272720:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078007.272722:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4992 00000100:00080000:3.0:1769078007.272725:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078007.272728:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078007.272731:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078007.272738:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616835200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078007.272743:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769078007.272747:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078007.272747:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078007.272749:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078007.272753:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4992 00000100:00080000:3.0:1769078007.272755:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078007.272757:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078007.272760:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078007.272766:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616835328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078007.272771:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078007.272774:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078007.272775:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078007.272778:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4992 00000100:00080000:3.0:1769078007.272780:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078007.272784:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078007/real 1769078007] req@ffff995b84559880 x1855007616834816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078007.272785:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078007.272790:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078007.272799:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616835456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078007.272805:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616834816/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078007.272820:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078007.272827:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078007.272829:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078007.272836:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4992 00000100:00080000:3.0:1769078007.272839:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078007.272840:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078007.272841:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078007.272846:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769078007.272847:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078007.272852:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616834816:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078007.272856:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616835584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078007.272872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078007.272888:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078007.272895:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078007/real 1769078007] req@ffff995b84558700 x1855007616834944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078007.272902:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616834944/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078007.272908:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078007.272910:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078007.272912:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616834944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078007.272915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078007.272924:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078007.272929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078007/real 1769078007] req@ffff995b8455b480 x1855007616835072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078007.272935:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616835072/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078007.272942:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078007.272944:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078007.272945:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078007.272949:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835200:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078007.272958:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078007.272963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078007/real 1769078007] req@ffff995b8455ad80 x1855007616835200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078007.272969:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616835200/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078007.272974:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078007.272976:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078007.272977:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835200:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078007.272980:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835328:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078007.272989:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078007.272994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078007/real 1769078007] req@ffff995b8455a300 x1855007616835328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078007.273000:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616835328/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078007.273005:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078007.273006:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078007.273008:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835328:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078007.273011:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616835456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078007.273019:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078007.273024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078007/real 1769078007] req@ffff995b84559c00 x1855007616835456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078007.273031:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616835456/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078007.273035:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078007.273037:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078007.273039:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616835456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078007.273042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616835584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078007.273049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078007.273055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078007/real 1769078007] req@ffff995b84558000 x1855007616835584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078007.273061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616835584/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078007.273065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078007.273067:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078007.273068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616835584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078007.976538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078007.976593:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078008.296532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078008.296532:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078008.296532:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078008.296562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078009.000548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078009.000601:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078009.320539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078009.320564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078009.320572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078009.321498:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078010.024544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078010.024600:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078010.344579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078010.344597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078010.344598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078010.344636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078011.048566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078011.048628:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078011.368552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078011.368552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078011.368555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078011.368595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078012.072528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078012.072605:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078012.392525:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078012.392527:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078012.392546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769078012.392565:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078012.392573:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078012.392577:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078012.392581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769078012.392588:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 4998 00000100:00080000:1.0:1769078012.392598:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078012.392605:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078012.392639:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616835712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078012.392662:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078012.392666:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078012.392668:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078012.392674:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4998 00000100:00080000:1.0:1769078012.392678:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078012.392683:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078012.392690:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616835712:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769078012.392694:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078012.392712:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616835840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078012.392724:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078012.392727:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078012.392729:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078012.392732:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4998 00000100:00080000:1.0:1769078012.392744:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078012.392747:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078012.392751:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769078012.392757:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078012.392764:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616835968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078012.392773:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078012.392775:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078012.392777:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078012.392780:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4998 00000100:00080000:1.0:1769078012.392782:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078012.392784:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078012.392788:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078012.392796:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616836096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078012.392809:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078012/real 1769078012] req@ffff995b868c6300 x1855007616835712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078012.392813:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078012.392820:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078012.392824:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078012.392829:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 4998 00000100:00080000:1.0:1769078012.392832:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078012.392837:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616835712/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078012.392837:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078012.392848:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769078012.392852:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769078012.392858:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616836224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078012.392862:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769078012.392872:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078012.392873:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616835712:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078012.392879:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078012.392882:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078012.392886:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4998 00000100:00080000:1.0:1769078012.392889:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078012.392892:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078012.392895:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078012.392900:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616836352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078012.392906:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078012.392908:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078012.392910:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078012.392915:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 4998 00000100:00100000:0.0:1769078012.392916:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835840:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078012.392917:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078012.392923:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078012.392929:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078012.392937:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616836480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078012.392957:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078012.392975:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078012/real 1769078012] req@ffff995b868c6d80 x1855007616835840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078012.392990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616835840/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078012.392999:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078012.393003:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078012.393007:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078012.393014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078012.393035:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078012.393045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078012/real 1769078012] req@ffff995b868c4a80 x1855007616835968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078012.393058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616835968/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078012.393066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078012.393068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078012.393070:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616835968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078012.393074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836096:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078012.393087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078012.393093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078012/real 1769078012] req@ffff995b868c6a00 x1855007616836096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078012.393101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616836096/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078012.393106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078012.393108:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078012.393110:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836096:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078012.393114:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836224:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078012.393123:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078012.393129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078012/real 1769078012] req@ffff995b868c7100 x1855007616836224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078012.393136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616836224/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078012.393146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078012.393147:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078012.393149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836224:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078012.393152:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616836352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078012.393162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078012.393167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078012/real 1769078012] req@ffff995b868c7480 x1855007616836352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078012.393174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616836352/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078012.393180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078012.393182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078012.393184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616836352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078012.393204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616836480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078012.393224:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078012.393236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078012/real 1769078012] req@ffff995b868c7800 x1855007616836480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078012.393247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616836480/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078012.393253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078012.393255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078012.393258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616836480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078013.096583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078013.096646:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078013.416531:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078013.416544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078013.416579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078013.416613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078014.120553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078014.120631:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078014.440537:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078014.440537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078014.440545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078014.441027:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078015.144572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078015.144648:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078015.464545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078015.464545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078015.464545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078015.464552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078016.168530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078016.168621:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078016.488569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078016.488578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078016.488580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078016.488608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078017.192567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078017.192635:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078017.512545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078017.512545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078017.512561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078017.512564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769078017.512602:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078017.512607:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078017.512612:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078017.512620:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5003 00000100:00080000:1.0:1769078017.512631:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078017.512638:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078017.512673:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616836608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078017.512695:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078017.512698:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078017.512700:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078017.512703:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5003 00000100:00080000:1.0:1769078017.512705:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078017.512709:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078017.512718:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078017.512729:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616836608:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078017.512734:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616836736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078017.512748:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078017.512751:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078017.512753:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078017.512757:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5003 00000100:00080000:1.0:1769078017.512762:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078017.512765:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078017.512770:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078017.512782:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616836864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078017.512793:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078017.512794:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078017.512800:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078017.512802:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078017.512808:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5003 00000100:00080000:1.0:1769078017.512812:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078017.512817:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078017.512823:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078017.512827:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078017/real 1769078017] req@ffff995b868c4700 x1855007616836608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078017.512838:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616836992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078017.512844:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616836608/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078017.512857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078017.512861:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078017.512868:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616836608:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078017.512871:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078017.512877:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078017.512879:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078017.512885:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5003 00000100:00080000:1.0:1769078017.512889:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078017.512893:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078017.512899:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078017.512900:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078017.512916:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616837120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078017.512929:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078017.512934:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078017.512935:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078017.512940:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078017.512951:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5003 00000100:00100000:0.0:1769078017.512956:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078017/real 1769078017] req@ffff995b868c5500 x1855007616836736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078017.512956:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078017.512964:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078017.512972:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078017.512983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616836736/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078017.512986:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616837248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078017.513015:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078017.513022:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078017.513026:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078017.513033:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5003 00000100:00080000:0.0:1769078017.513036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078017.513037:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078017.513043:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078017.513047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:1.0:1769078017.513048:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078017.513053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078017.513058:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616837376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078017.513063:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078017.513091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078017.513101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078017/real 1769078017] req@ffff995b868c5c00 x1855007616836864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078017.513109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616836864/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078017.513117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078017.513119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078017.513121:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078017.513125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836992:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078017.513137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078017.513142:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078017/real 1769078017] req@ffff995b868c7800 x1855007616836992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078017.513150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616836992/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078017.513155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078017.513157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078017.513160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616836992:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078017.513210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616837120:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078017.513226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078017.513234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078017/real 1769078017] req@ffff995b868c7480 x1855007616837120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078017.513242:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616837120/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078017.513248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078017.513250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078017.513253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616837120:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078017.513256:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616837248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078017.513266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078017.513272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078017/real 1769078017] req@ffff995b868c7100 x1855007616837248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078017.513279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616837248/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078017.513285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078017.513286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078017.513288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616837248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078017.513291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616837376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078017.513306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078017.513317:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078017/real 1769078017] req@ffff995b868c6a00 x1855007616837376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078017.513329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616837376/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078017.513339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078017.513343:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078017.513347:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616837376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078018.216541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078018.216599:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078018.536566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078018.536566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078018.536613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078018.536622:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078019.240563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078019.240619:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078019.560546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078019.560631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078019.560635:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078019.560638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078020.264524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078020.264584:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078020.584562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078020.584563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078020.584570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078020.584586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078021.288566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078021.288627:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078021.608567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078021.608568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078021.608569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078021.608621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078022.312561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078022.312626:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078022.632603:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078022.632603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078022.632603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078022.632615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078022.632654:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078022.632660:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078022.632663:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078022.632675:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5008 00000100:00080000:1.0:1769078022.632686:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078022.632694:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078022.632732:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616837504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078022.632765:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078022.632768:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078022.632769:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078022.632773:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5008 00000100:00080000:1.0:1769078022.632775:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078022.632778:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078022.632786:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078022.632797:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616837632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078022.632804:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078022.632808:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078022.632809:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078022.632811:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5008 00000100:00080000:1.0:1769078022.632813:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078022.632816:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078022.632819:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078022.632826:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616837504:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078022.632827:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616837760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078022.632842:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078022.632844:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078022.632845:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078022.632848:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5008 00000100:00080000:1.0:1769078022.632850:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078022.632853:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078022.632856:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078022.632867:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616837888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078022.632873:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078022.632875:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078022.632877:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078022.632879:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5008 00000100:00080000:1.0:1769078022.632894:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078022.632896:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078022.632899:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769078022.632905:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078022.632909:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616838016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078022.632917:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078022.632920:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078022.632922:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078022.632930:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5008 00000100:00080000:1.0:1769078022.632933:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078022.632936:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078022.632940:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078022.632948:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616838144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078022.632959:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078022.632963:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078022.632964:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078022/real 1769078022] req@ffff995b868c4a80 x1855007616837504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078022.632965:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078022.632971:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5008 00000100:00080000:1.0:1769078022.632975:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078022.632980:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078022.632986:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078022.632996:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616838272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078022.633001:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616837504/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078022.633012:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078022.633016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078022.633021:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616837504:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078022.633027:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616837632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078022.633047:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078022.633057:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078022/real 1769078022] req@ffff995b868c6d80 x1855007616837632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078022.633065:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616837632/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078022.633070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078022.633073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078022.633075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616837632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078022.633120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616837760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078022.633135:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078022.633143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078022/real 1769078022] req@ffff995b868c6300 x1855007616837760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078022.633151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616837760/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078022.633159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078022.633161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078022.633163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616837760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078022.633167:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616837888:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078022.633177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078022.633184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078022/real 1769078022] req@ffff995b868c6a00 x1855007616837888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078022.633192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616837888/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078022.633198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078022.633200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078022.633201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616837888:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078022.633205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838016:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078022.633218:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078022.633229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078022/real 1769078022] req@ffff995b868c7100 x1855007616838016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078022.633253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616838016/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078022.633268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078022.633272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078022.633276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838016:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078022.633282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616838144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078022.633298:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078022.633306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078022/real 1769078022] req@ffff995b868c7480 x1855007616838144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078022.633314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616838144/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078022.633321:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078022.633323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078022.633325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616838144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078022.633328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616838272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078022.633338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078022.633343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078022/real 1769078022] req@ffff995b868c7800 x1855007616838272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078022.633350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616838272/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078022.633356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078022.633357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078022.633359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616838272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078023.336541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078023.336644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078023.656619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078023.656626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078023.656626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078023.656658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078024.360588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078024.360674:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078024.680616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078024.680617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078024.680617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078024.680621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078025.384565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078025.384657:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078025.704589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078025.704589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078025.704614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078025.704643:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078026.408605:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078026.408675:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078026.728607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078026.728626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078026.728653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078026.728661:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078027.432600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078027.432670:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078027.752573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078027.752573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078027.752588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078027.752613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078027.752630:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078027.752637:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078027.752642:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078027.752651:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5013 00000100:00080000:3.0:1769078027.752663:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078027.752670:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078027.752709:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616838400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078027.752732:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078027.752734:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078027.752736:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078027.752739:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5013 00000100:00080000:3.0:1769078027.752747:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078027.752750:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078027.752758:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078027.752767:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616838400:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078027.752770:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616838528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078027.752782:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078027.752785:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078027.752787:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078027.752790:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5013 00000100:00080000:3.0:1769078027.752794:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078027.752798:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078027.752803:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078027.752814:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616838656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078027.752825:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078027.752827:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078027.752831:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078027.752834:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078027.752838:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5013 00000100:00080000:3.0:1769078027.752842:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078027.752845:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078027.752849:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078027.752860:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616838784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078027.752862:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078027/real 1769078027] req@ffff995b84559500 x1855007616838400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078027.752879:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078027.752882:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078027.752883:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078027.752886:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5013 00000100:00100000:0.0:1769078027.752888:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616838400/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078027.752889:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078027.752897:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078027.752901:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078027.752904:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000008:00100000:3.0:1769078027.752905:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078027.752909:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616838400:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078027.752916:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616838912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078027.752922:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078027.752925:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078027.752926:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078027.752928:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5013 00000100:00100000:0.0:1769078027.752931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838528:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078027.752933:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078027.752939:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078027.752942:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078027.752951:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616839040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078027.752957:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078027.752961:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078027.752965:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078027.752966:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078027.752969:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5013 00000100:00080000:3.0:1769078027.752972:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078027.752976:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078027.752977:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078027/real 1769078027] req@ffff995b8455bb80 x1855007616838528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078027.752979:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078027.752988:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616839168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078027.753006:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616838528/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078027.753026:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078027.753028:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078027.753031:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078027.753035:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078027.753052:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078027.753063:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078027/real 1769078027] req@ffff995b84558a80 x1855007616838656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078027.753075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616838656/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078027.753091:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078027.753094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078027.753097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078027.753102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838784:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078027.753116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078027.753123:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078027/real 1769078027] req@ffff995b8455a680 x1855007616838784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078027.753129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616838784/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078027.753134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078027.753136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078027.753138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838784:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078027.753141:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838912:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078027.753150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078027.753154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078027/real 1769078027] req@ffff995b8455b100 x1855007616838912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078027.753161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616838912/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078027.753166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078027.753168:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078027.753170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616838912:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078027.753181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616839040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078027.753190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078027.753196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078027/real 1769078027] req@ffff995b8455aa00 x1855007616839040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078027.753202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616839040/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078027.753207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078027.753209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078027.753210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616839040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078027.753213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616839168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078027.753221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078027.753226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078027/real 1769078027] req@ffff995b84559500 x1855007616839168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078027.753233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616839168/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078027.753238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078027.753239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078027.753241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616839168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078028.456548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078028.456607:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078028.776554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078028.776554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078028.776601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078028.776622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078029.480575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078029.480676:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078029.800615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078029.800632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078029.800775:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078029.800789:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078030.504559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078030.504618:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078030.824632:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078030.824632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078030.824635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078030.824641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078031.528558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078031.528614:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078031.848599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078031.848601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078031.848601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078031.848610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078032.552558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078032.552631:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078032.872600:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078032.872610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078032.872611:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078032.872618:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078032.872627:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5018 00000100:00080000:3.0:1769078032.872639:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078032.872647:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078032.872698:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616839296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078032.872700:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078032.872730:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078032.872734:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078032.872736:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078032.872741:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5018 00000100:00080000:3.0:1769078032.872745:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078032.872749:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078032.872759:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078032.872773:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616839424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078032.872782:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078032.872785:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078032.872787:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078032.872791:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5018 00000100:00080000:3.0:1769078032.872795:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078032.872800:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078032.872807:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078032.872823:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616839552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078032.872830:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078032.872833:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078032.872834:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078032.872838:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5018 00000100:00080000:3.0:1769078032.872841:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078032.872845:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078032.872849:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078032.872857:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616839680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078032.872863:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078032.872865:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078032.872866:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078032.872867:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5018 00000100:00080000:3.0:1769078032.872869:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078032.872871:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078032.872873:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078032.872878:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616839808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078032.872885:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078032.872888:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078032.872891:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078032.872972:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5018 00000100:00080000:3.0:1769078032.872977:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078032.872984:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078032.872991:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078032.873005:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616839936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078032.873156:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078032.873160:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078032.873163:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078032.873171:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5018 00000100:00100000:1.0:1769078032.873287:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078032.873318:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078032.873326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616839296:192.168.204.132@tcp:250:kworker.0 00000800:00000100:0.0:1769078032.873397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078032.873439:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078032/real 1769078032] req@ffff995b84558000 x1855007616839296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.873486:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616839296/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.873496:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.873501:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078032.873504:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616839296:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078032.873512:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616839424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078032.873533:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078032.873545:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078032/real 1769078032] req@ffff995b84559c00 x1855007616839424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.873558:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616839424/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.873565:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.873568:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078032.873572:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616839424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078032.873578:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616839552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078032.873596:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078032.873607:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078032/real 1769078032] req@ffff995b8455a300 x1855007616839552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.873617:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616839552/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.873626:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.873629:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078032.873632:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616839552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078032.873638:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616839680:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078032.873655:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078032.873665:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078032/real 1769078032] req@ffff995b8455ad80 x1855007616839680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.873676:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616839680/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.873684:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.873687:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078032.873690:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616839680:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078032.873695:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616839808:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078032.873710:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078032.873720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078032/real 1769078032] req@ffff995b8455b480 x1855007616839808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.873731:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616839808/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.873740:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.873743:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078032.873746:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616839808:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078032.873751:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616839936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078032.873768:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078032.873777:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078032/real 1769078032] req@ffff995b84558700 x1855007616839936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.873787:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616839936/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.873795:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.873798:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078032.873801:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616839936:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078032.873968:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078032.873973:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078032.873991:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078032.874009:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616840064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078032.874076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616840064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078032.874101:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078032.874115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078032/real 1769078032] req@ffff995b84559880 x1855007616840064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.874128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616840064/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.874138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.874142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078032.874145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616840064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078033.576551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078033.576600:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078033.896566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078033.896566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078033.896594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078033.896619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078034.600566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078034.600623:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078034.920552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078034.920552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078034.920556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078034.920587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078035.624551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078035.624598:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078035.944556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078035.944556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078035.944556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078035.944587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078036.648549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078036.648618:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078036.968521:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078036.968524:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078036.968530:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078036.968555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078037.672547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078037.672620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078037.992562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078037.992563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078037.992577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078037.992581:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078037.992588:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078037.992594:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078037.992602:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5023 00000100:00080000:3.0:1769078037.992611:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078037.992617:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078037.992630:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078037.992655:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616840192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078037.992676:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078037.992678:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078037.992680:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078037.992682:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5023 00000100:00080000:3.0:1769078037.992685:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078037.992687:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078037.992696:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078037.992705:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616840320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078037.992710:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616840192:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078037.992714:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078037.992718:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078037.992719:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078037.992724:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5023 00000100:00080000:3.0:1769078037.992726:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078037.992730:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078037.992734:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078037.992746:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616840448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078037.992756:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078037.992759:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078037.992761:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078037.992764:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5023 00000100:00080000:3.0:1769078037.992768:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078037.992772:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078037.992776:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769078037.992779:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078037.992787:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616840576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078037.992798:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078037.992801:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078037.992803:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078037.992807:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5023 00000100:00080000:3.0:1769078037.992812:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078037.992816:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078037.992820:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078037/real 1769078037] req@ffff995b84559880 x1855007616840192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078037.992823:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078037.992850:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616840704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078037.992857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616840192/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078037.992863:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078037.992872:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078037.992874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078037.992875:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078037.992884:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769078037.992885:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5023 00000100:00080000:3.0:1769078037.992892:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078037.992894:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616840192:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078037.992898:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078037.992905:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078037.992915:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616840832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078037.992923:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078037.992927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616840320:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078037.992927:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078037.992928:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078037.992934:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5023 00000100:00080000:3.0:1769078037.992937:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078037.992939:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078037.992942:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078037.992950:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616840960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078037.992964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078037.992980:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078037/real 1769078037] req@ffff995b84558700 x1855007616840320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078037.993003:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616840320/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078037.993012:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078037.993015:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078037.993018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616840320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078037.993024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616840448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078037.993040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078037.993049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078037/real 1769078037] req@ffff995b8455b480 x1855007616840448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078037.993065:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616840448/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078037.993075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078037.993078:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078037.993081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616840448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078037.993087:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616840576:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078037.993101:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078037.993109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078037/real 1769078037] req@ffff995b8455ad80 x1855007616840576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078037.993119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616840576/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078037.993134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078037.993137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078037.993140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616840576:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078037.993145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616840704:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078037.993168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078037.993176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078037/real 1769078037] req@ffff995b8455a300 x1855007616840704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078037.993187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616840704/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078037.993195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078037.993198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078037.993201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616840704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078037.993216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616840832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078037.993230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078037.993239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078037/real 1769078037] req@ffff995b84559c00 x1855007616840832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078037.993249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616840832/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078037.993256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078037.993259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078037.993263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616840832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078037.993268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616840960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078037.993281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078037.993289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078037/real 1769078037] req@ffff995b84558000 x1855007616840960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078037.993300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616840960/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078037.993308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078037.993311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078037.993314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616840960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078038.696552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078038.696616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078039.016539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078039.016539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078039.016539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078039.016544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078039.720552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078039.720604:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078040.040569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078040.040570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078040.040570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078040.040626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769078040.744625:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078040.744661:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078041.064541:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078041.064543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078041.064549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078041.064557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078041.768513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078041.768592:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078042.088555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078042.088555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078042.088555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078042.088574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078042.792578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078042.792648:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078043.112551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078043.112552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078043.112552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078043.112560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078043.112589:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078043.112596:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078043.112601:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078043.112609:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5028 00000100:00080000:0.0:1769078043.112620:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078043.112635:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078043.112678:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616841088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078043.112704:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078043.112706:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078043.112708:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078043.112710:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5028 00000100:00080000:0.0:1769078043.112713:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078043.112717:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078043.112725:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078043.112735:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616841216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078043.112738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616841088:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078043.112744:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078043.112747:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078043.112749:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078043.112762:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5028 00000100:00080000:0.0:1769078043.112765:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078043.112768:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078043.112771:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078043.112781:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616841344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078043.112788:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078043.112790:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078043.112791:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078043.112794:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5028 00000100:00080000:0.0:1769078043.112797:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078043.112799:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078043.112802:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078043.112809:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616841472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078043.112815:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769078043.112819:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078043.112820:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078043.112824:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078043.112827:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5028 00000100:00080000:0.0:1769078043.112829:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078043.112831:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078043.112835:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078043.112841:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616841600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078043.112847:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078043.112849:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078043.112850:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078043.112854:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5028 00000100:00100000:2.0:1769078043.112855:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078043/real 1769078043] req@ffff995b84643800 x1855007616841088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078043.112857:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078043.112862:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078043.112869:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078043.112878:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616841728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078043.112880:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616841088/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078043.112897:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078043.112900:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078043.112900:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078043.112901:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078043.112906:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078043.112908:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5028 00000100:00080000:0.0:1769078043.112912:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769078043.112914:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616841088:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078043.112916:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078043.112921:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078043.112928:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616841856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078043.112946:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616841216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078043.112970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078043.112980:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078043/real 1769078043] req@ffff995b84641180 x1855007616841216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078043.112989:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616841216/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078043.112994:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078043.112997:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078043.112998:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616841216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078043.113007:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616841344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078043.113030:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078043.113041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078043/real 1769078043] req@ffff995b84641500 x1855007616841344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078043.113054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616841344/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078043.113062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078043.113065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078043.113068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616841344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078043.113074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616841472:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078043.113094:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078043.113105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078043/real 1769078043] req@ffff995b84640e00 x1855007616841472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078043.113116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616841472/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078043.113122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078043.113124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078043.113126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616841472:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078043.113130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616841600:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078043.113143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078043.113149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078043/real 1769078043] req@ffff995b84642680 x1855007616841600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078043.113156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616841600/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078043.113162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078043.113164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078043.113165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616841600:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078043.113168:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616841728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078043.113176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078043.113181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078043/real 1769078043] req@ffff995b84640000 x1855007616841728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078043.113188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616841728/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078043.113192:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078043.113194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078043.113195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616841728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078043.113198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616841856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078043.113206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078043.113211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078043/real 1769078043] req@ffff995b84640380 x1855007616841856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078043.113217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616841856/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078043.113225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078043.113227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078043.113228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616841856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078043.816560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078043.816649:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078044.136548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078044.136548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078044.136548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078044.136559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078044.840563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078044.840641:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078045.160584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078045.160584:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078045.160584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078045.160584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078045.864548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078045.864607:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078046.184552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078046.184558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078046.184559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078046.184562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078046.888572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078046.888630:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078047.208559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078047.208560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078047.208721:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078047.208735:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078047.912598:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078047.912634:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078048.232558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078048.232558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078048.232559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078048.232603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078048.232628:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078048.232634:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078048.232638:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078048.232647:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5033 00000100:00080000:2.0:1769078048.232657:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078048.232662:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078048.232716:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616841984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078048.232738:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078048.232741:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078048.232742:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078048.232745:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5033 00000100:00080000:2.0:1769078048.232748:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078048.232751:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078048.232759:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078048.232768:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616842112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078048.232774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616841984:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078048.232777:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078048.232781:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078048.232782:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078048.232785:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5033 00000100:00080000:2.0:1769078048.232787:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078048.232789:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078048.232792:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078048.232801:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616842240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078048.232806:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078048.232808:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078048.232810:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078048.232812:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5033 00000100:00080000:2.0:1769078048.232814:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078048.232816:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078048.232819:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078048.232826:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616842368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078048.232833:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078048.232841:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078048.232843:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078048.232845:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078048.232848:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5033 00000100:00080000:2.0:1769078048.232851:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078048.232852:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078048.232855:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078048.232862:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616842496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078048.232867:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078048.232869:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078048/real 1769078048] req@ffff995b85ac9c00 x1855007616841984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078048.232871:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078048.232873:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078048.232880:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5033 00000100:00080000:2.0:1769078048.232883:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078048.232886:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078048.232889:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078048.232890:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616841984/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078048.232897:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616842624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078048.232900:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078048.232909:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769078048.232909:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078048.232914:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078048.232917:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616841984:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078048.232917:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078048.232924:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5033 00000100:00080000:2.0:1769078048.232927:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078048.232929:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078048.232931:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078048.232937:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616842752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078048.232942:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616842112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078048.232959:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078048.232967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078048/real 1769078048] req@ffff995b85acb100 x1855007616842112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078048.232975:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616842112/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078048.232980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078048.232982:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078048.232984:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616842112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078048.232987:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616842240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078048.232997:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078048.233002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078048/real 1769078048] req@ffff995b85acaa00 x1855007616842240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078048.233009:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616842240/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078048.233015:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078048.233017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078048.233018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616842240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078048.233021:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616842368:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078048.233030:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078048.233036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078048/real 1769078048] req@ffff995b85ac9880 x1855007616842368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078048.233043:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616842368/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078048.233048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078048.233049:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078048.233051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616842368:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078048.233054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616842496:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078048.233062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078048.233066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078048/real 1769078048] req@ffff995b85acbb80 x1855007616842496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078048.233073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616842496/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078048.233078:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078048.233080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078048.233081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616842496:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078048.233084:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616842624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078048.233092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078048.233097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078048/real 1769078048] req@ffff995b85acb800 x1855007616842624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078048.233108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616842624/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078048.233113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078048.233114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078048.233116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616842624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078048.233126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616842752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078048.233134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078048.233139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078048/real 1769078048] req@ffff995b85ac8a80 x1855007616842752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078048.233154:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616842752/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078048.233159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078048.233161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078048.233163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616842752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078048.936513:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078048.936616:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078049.256572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078049.256572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078049.256585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078049.256603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078049.960558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078049.960617:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078050.280536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078050.280543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078050.280553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078050.280553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078050.984557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078050.984601:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078051.304555:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078051.304555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078051.304555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078051.304741:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078052.008527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078052.008579:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078052.328558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078052.328558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078052.328578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078052.328591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078053.032550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078053.032622:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078053.352556:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078053.352559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769078053.352567:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078053.352573:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078053.352575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078053.352599:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078053.352604:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5039 00000100:00100000:1.0:1769078053.352614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078053.352616:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078053.352625:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078053.352663:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616842880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078053.352686:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078053.352689:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078053.352691:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078053.352694:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5039 00000100:00080000:2.0:1769078053.352698:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078053.352701:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078053.352710:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078053.352720:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616843008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078053.352730:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616842880:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078053.352730:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078053.352737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078053.352740:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078053.352744:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5039 00000100:00080000:2.0:1769078053.352746:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078053.352748:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078053.352752:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078053.352762:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616843136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078053.352769:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078053.352772:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078053.352774:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078053.352776:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5039 00000100:00080000:2.0:1769078053.352779:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078053.352781:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078053.352784:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078053.352791:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616843264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078053.352802:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078053.352808:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078053.352808:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078053.352812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078053.352815:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5039 00000100:00080000:2.0:1769078053.352817:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078053.352820:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078053.352825:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078053.352837:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616843392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078053.352845:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078053.352849:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078053.352851:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078053.352856:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5039 00000100:00080000:2.0:1769078053.352860:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078053.352865:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078053/real 1769078053] req@ffff995b85ac8380 x1855007616842880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078053.352869:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078053.352878:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078053.352893:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616842880/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078053.352894:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616843520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078053.352925:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078053.352929:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078053.352930:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078053.352931:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078053.352939:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5039 00000100:00080000:0.0:1769078053.352943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769078053.352944:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078053.352952:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078053.352954:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616842880:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769078053.352956:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078053.352965:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616843648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078053.352997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078053.353025:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078053.353038:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078053/real 1769078053] req@ffff995b85aca300 x1855007616843008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078053.353051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616843008/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078053.353060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078053.353064:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078053.353068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078053.353074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078053.353099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078053.353111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078053/real 1769078053] req@ffff995b85ac8a80 x1855007616843136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078053.353124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616843136/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078053.353135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078053.353138:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078053.353141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078053.353147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843264:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078053.353167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078053.353178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078053/real 1769078053] req@ffff995b85acb800 x1855007616843264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078053.353199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616843264/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078053.353212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078053.353216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078053.353219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843264:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078053.353226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843392:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078053.353244:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078053.353256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078053/real 1769078053] req@ffff995b85acbb80 x1855007616843392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078053.353287:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616843392/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078053.353297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078053.353300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078053.353304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078053.353310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616843520:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078053.353331:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078053.353342:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078053/real 1769078053] req@ffff995b85ac9880 x1855007616843520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078053.353354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616843520/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078053.353373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078053.353376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078053.353380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616843520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078053.353386:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616843648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078053.353406:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078053.353417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078053/real 1769078053] req@ffff995b85acaa00 x1855007616843648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078053.353429:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616843648/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078053.353443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078053.353446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078053.353449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616843648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078054.056580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078054.056649:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078054.376579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078054.376580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078054.376590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078054.376616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078055.080580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078055.080652:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078055.400581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078055.400581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078055.400586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078055.400601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078056.104575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078056.104637:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078056.424539:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078056.424560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078056.424566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078056.424618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078057.128567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078057.128683:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078057.448573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078057.448573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078057.448592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078057.448601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078058.152567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078058.152641:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078058.472547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078058.472547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078058.472550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078058.472611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078058.472624:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078058.472631:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078058.472635:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078058.472644:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5044 00000100:00080000:2.0:1769078058.472654:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078058.472659:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078058.472696:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616843776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078058.472718:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078058.472721:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078058.472723:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078058.472725:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5044 00000100:00080000:2.0:1769078058.472728:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078058.472731:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078058.472740:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078058.472752:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616843776:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078058.472758:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616843904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078058.472772:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078058.472777:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078058.472779:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078058.472784:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5044 00000100:00080000:2.0:1769078058.472789:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078058.472793:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078058.472798:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078058.472814:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616844032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078058.472816:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078058.472826:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078058.472829:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078058.472831:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078058.472835:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5044 00000100:00080000:2.0:1769078058.472838:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078058.472842:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078058.472847:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078058.472853:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078058/real 1769078058] req@ffff995b85acb100 x1855007616843776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078058.472858:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616844160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078058.472875:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078058.472878:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1769078058.472881:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616843776/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078058.472881:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078058.472890:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5044 00000100:00080000:2.0:1769078058.472896:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078058.472901:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078058.472902:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769078058.472904:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769078058.472908:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078058.472912:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616843776:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078058.472915:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616844288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078058.472923:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078058.472925:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078058.472927:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078058.472930:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5044 00000100:00080000:2.0:1769078058.472933:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078058.472937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843904:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078058.472938:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078058.472946:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078058.472956:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616844416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078058.472964:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078058.472965:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078058.472970:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078058.472973:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078058.472977:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5044 00000100:00080000:2.0:1769078058.472980:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078058.472981:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078058/real 1769078058] req@ffff995b85ac9c00 x1855007616843904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078058.472985:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078058.472992:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078058.473000:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616843904/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078058.473005:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616844544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078058.473016:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078058.473023:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078058.473027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616843904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078058.473032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078058.473046:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078058.473053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078058/real 1769078058] req@ffff995b85acaa00 x1855007616844032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078058.473061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616844032/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078058.473067:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078058.473069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078058.473071:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078058.473075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844160:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078058.473089:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078058.473099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078058/real 1769078058] req@ffff995b85ac9880 x1855007616844160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078058.473111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616844160/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078058.473128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078058.473131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078058.473135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844160:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078058.473142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844288:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078058.473164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078058.473175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078058/real 1769078058] req@ffff995b85acbb80 x1855007616844288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078058.473188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616844288/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078058.473198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078058.473202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078058.473206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078058.473222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616844416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078058.473251:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078058.473259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078058/real 1769078058] req@ffff995b85acb800 x1855007616844416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078058.473267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616844416/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078058.473273:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078058.473275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078058.473278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616844416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078058.473282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616844544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078058.473292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078058.473297:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078058/real 1769078058] req@ffff995b85ac8a80 x1855007616844544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078058.473305:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616844544/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078058.473310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078058.473312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078058.473314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616844544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078059.176548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078059.176623:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078059.496588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078059.496593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078059.496618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078059.496638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078060.200570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078060.200646:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078060.520578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078060.520578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078060.520610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078060.520639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078061.224540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078061.224600:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078061.544587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078061.544605:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078061.544626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078061.544655:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078062.248564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078062.248628:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078062.568569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078062.568569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078062.568573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078062.568595:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078063.272528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078063.272586:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078063.592572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078063.592574:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078063.592575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078063.592578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078063.592587:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078063.592592:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078063.592600:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5049 00000100:00080000:2.0:1769078063.592611:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078063.592618:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078063.592619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078063.592659:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616844672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078063.592682:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078063.592685:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078063.592687:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078063.592690:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5049 00000100:00080000:2.0:1769078063.592692:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078063.592696:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078063.592705:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078063.592716:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616844800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078063.592717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616844672:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078063.592733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078063.592736:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078063.592737:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078063.592740:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5049 00000100:00080000:2.0:1769078063.592743:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078063.592745:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078063.592748:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078063.592757:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616844928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078063.592764:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078063.592767:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078063.592769:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078063.592771:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5049 00000100:00080000:2.0:1769078063.592773:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078063.592775:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078063.592778:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078063.592786:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616845056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078063.592792:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078063.592795:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078063.592796:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078063.592799:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5049 00000100:00080000:2.0:1769078063.592802:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078063.592806:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078063.592809:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078063.592819:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616845184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078063.592828:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078063.592831:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844800:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078063.592832:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078063.592836:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078063.592842:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5049 00000100:00080000:2.0:1769078063.592844:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078063.592847:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078063.592850:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078063.592860:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616845312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078063.592867:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844928:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078063.592868:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078063.592876:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078063.592879:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078063.592883:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5049 00000100:00080000:2.0:1769078063.592887:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078063.592890:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078063.592893:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078063.592894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078063.592920:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616845440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078063.592930:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078063.592953:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616845312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078063.592969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616845440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078064.296564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078064.296626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078064.616582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078064.616582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078064.616630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078064.616662:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078065.320523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078065.320595:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078065.640568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078065.640568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078065.640580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078065.640596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078066.344534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078066.344614:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078066.664598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078066.664598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078066.664725:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078066.664742:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000400:00000100:1.0:1769078066.664841:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769078066.664910:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769078066.664935:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078066.664987:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078066.665002:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078066.665011:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078066.665020:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078066.665029:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078066.665039:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769078066.665196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078063/real 1769078066] req@ffff995b85acaa00 x1855007616845440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078066.665228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616845440/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078066.665243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078066.665249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078066.665264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616845440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078066.665274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078063/real 1769078066] req@ffff995b85acbb80 x1855007616845184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078066.665290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616845184/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078066.665300:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078066.665305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078066.665308:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078066.665316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078063/real 1769078066] req@ffff995b85ac9880 x1855007616845312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078066.665327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616845312/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078066.665335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078066.665338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078066.665342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616845312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078066.665348:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078063/real 1769078066] req@ffff995b85ac8380 x1855007616844800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078066.665392:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616844800/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078066.665402:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078066.665407:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078066.665411:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078066.665418:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078063/real 1769078066] req@ffff995b85ac8a80 x1855007616844928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078066.665431:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616844928/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078066.665445:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078066.665450:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078066.665481:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616844928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078066.665493:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078063/real 1769078066] req@ffff995b85acb800 x1855007616845056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078066.665513:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616845056/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078066.665525:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078066.665531:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078066.665535:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845056:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078066.665545:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078063/real 1769078066] req@ffff995b85aca300 x1855007616844672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078066.665563:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616844672/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078066.665572:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078066.665578:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078066.665582:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616844672:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078067.368552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078067.368616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078067.688548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078067.688548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078067.688559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078067.688596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078068.392564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078068.392664:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078068.712572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078068.712577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078068.712720:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078068.712728:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078068.712736:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078068.712742:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078068.712754:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5054 00000100:00080000:3.0:1769078068.712766:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078068.712775:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078068.712789:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078068.712833:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616845568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078068.712873:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078068.712878:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078068.712880:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078068.712885:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5054 00000100:00080000:3.0:1769078068.712890:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078068.712900:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078068.712913:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078068.712917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616845568:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078068.712931:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616845696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078068.712946:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078068.712950:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078068.712952:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078068.712957:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5054 00000100:00080000:3.0:1769078068.712962:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078068.712966:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078068.712973:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078068.712984:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616845824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078068.712995:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078068.712998:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1769078068.713003:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078068.713003:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078068.713012:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5054 00000100:00080000:3.0:1769078068.713017:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078068.713022:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078068.713029:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078068.713043:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616845952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078068.713054:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078068.713055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078068/real 1769078068] req@ffff995b84559500 x1855007616845568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078068.713061:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078068.713066:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078068.713075:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5054 00000100:00080000:3.0:1769078068.713079:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078068.713084:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078068.713090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616845568/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078068.713090:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769078068.713106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769078068.713108:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616846080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078068.713118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769078068.713124:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078068.713131:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769078068.713133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616845568:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078068.713134:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078068.713142:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5054 00000100:00080000:3.0:1769078068.713148:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078068.713153:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078068.713159:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078068.713170:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616846208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078068.713175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845696:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078068.713190:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078068.713195:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078068.713198:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078068.713204:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5054 00000100:00080000:3.0:1769078068.713209:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078068.713214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078068.713218:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078068.713225:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078068.713235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078068/real 1769078068] req@ffff995b8455aa00 x1855007616845696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078068.713241:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616846336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078068.713265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616845696/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078068.713278:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078068.713283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078068.713287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078068.713295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078068.713318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078068.713355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078068/real 1769078068] req@ffff995b8455b100 x1855007616845824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078068.713368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616845824/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078068.713387:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078068.713391:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078068.713399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078068.713407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845952:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078068.713429:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078068.713441:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078068/real 1769078068] req@ffff995b8455a680 x1855007616845952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078068.713474:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616845952/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078068.713486:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078068.713489:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078068.713493:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616845952:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078068.713499:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846080:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078068.713522:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078068.713533:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078068/real 1769078068] req@ffff995b84558a80 x1855007616846080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078068.713545:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616846080/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078068.713554:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078068.713557:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078068.713561:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078068.713580:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616846208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078068.713601:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078068.713613:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078068/real 1769078068] req@ffff995b8455bb80 x1855007616846208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078068.713626:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616846208/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078068.713635:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078068.713638:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078068.713642:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616846208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078068.713648:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616846336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078068.713668:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078068.713680:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078068/real 1769078068] req@ffff995b84559500 x1855007616846336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078068.713693:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616846336/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078068.713702:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078068.713706:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078068.713709:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616846336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078069.416560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078069.416666:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078069.736571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078069.736571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078069.736589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078069.736612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078070.440599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078070.440667:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078070.760610:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078070.760616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078070.760618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078070.760651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078071.464590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078071.464633:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078071.784594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078071.784595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078071.784597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078071.784607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078072.488574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078072.488642:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078072.808609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078072.808609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078072.808613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078072.808651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078073.512579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078073.512665:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078073.832554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078073.832568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078073.832573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078073.832600:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078073.832607:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078073.832612:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078073.832625:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5059 00000100:00100000:1.0:1769078073.832631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078073.832636:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078073.832645:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078073.832682:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616846464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078073.832704:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078073.832707:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078073.832708:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078073.832711:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5059 00000100:00080000:3.0:1769078073.832713:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078073.832716:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078073.832725:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078073.832741:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616846592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078073.832750:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616846464:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078073.832764:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078073.832767:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078073.832769:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078073.832773:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5059 00000100:00080000:3.0:1769078073.832777:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078073.832780:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078073.832785:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078073.832795:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616846720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078073.832802:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078073.832804:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078073.832805:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078073.832809:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5059 00000100:00080000:3.0:1769078073.832815:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078073.832817:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078073.832821:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078073.832827:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616846848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078073.832834:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078073.832837:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078073.832839:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078073.832843:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5059 00000800:00000100:0.0:1769078073.832851:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078073.832855:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078073.832860:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078073.832865:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078073.832878:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616846976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078073.832884:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078073.832886:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078073.832888:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078073.832891:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5059 00000100:00080000:3.0:1769078073.832894:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078073.832897:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078073.832900:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078073.832905:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078073/real 1769078073] req@ffff995b84558000 x1855007616846464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078073.832913:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616847104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078073.832938:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616846464/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078073.832939:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078073.832960:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078073.832965:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078073.832966:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078073.832973:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769078073.832979:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5059 00000100:00100000:0.0:1769078073.832984:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616846464:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078073.832990:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078073.832998:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078073.833003:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078073.833015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078073.833023:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616847232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078073.833040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078073.833054:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078073/real 1769078073] req@ffff995b84559c00 x1855007616846592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078073.833063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616846592/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078073.833070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078073.833072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078073.833075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078073.833080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078073.833091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078073.833097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078073/real 1769078073] req@ffff995b8455a300 x1855007616846720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078073.833104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616846720/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078073.833111:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078073.833113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078073.833116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078073.833120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846848:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078073.833132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078073.833137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078073/real 1769078073] req@ffff995b8455ad80 x1855007616846848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078073.833145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616846848/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078073.833151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078073.833153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078073.833155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846848:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078073.833158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078073.833167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078073.833174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078073/real 1769078073] req@ffff995b8455b480 x1855007616846976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078073.833181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616846976/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078073.833187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078073.833189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078073.833191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616846976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078073.833194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616847104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078073.833204:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078073.833210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078073/real 1769078073] req@ffff995b84558700 x1855007616847104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078073.833216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616847104/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078073.833221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078073.833224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078073.833225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616847104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078073.833257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616847232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078073.833271:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078073.833278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078073/real 1769078073] req@ffff995b84559880 x1855007616847232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078073.833286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616847232/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078073.833292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078073.833294:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078073.833296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616847232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078074.536581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078074.536634:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078074.856582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078074.856582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078074.856615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078074.856650:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078075.560562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078075.560628:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078075.880601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078075.880623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078075.880765:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078075.880790:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078076.584564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078076.584650:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078076.904771:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078076.904776:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078076.904778:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078076.905035:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078077.608584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078077.608653:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078077.928511:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078077.928530:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078077.928563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078077.928584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078078.632557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078078.632611:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078078.952553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078078.952556:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078078.952565:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078078.952569:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078078.952571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078078.952576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769078078.952579:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5064 00000100:00080000:1.0:1769078078.952592:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078078.952594:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769078078.952600:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078078.952636:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616847360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078078.952656:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078078.952659:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078078.952661:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078078.952665:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5064 00000100:00080000:1.0:1769078078.952669:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078078.952674:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078078.952684:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078078.952694:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616847360:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078078.952701:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616847488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078078.952713:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078078.952717:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078078.952719:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078078.952722:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5064 00000100:00080000:1.0:1769078078.952725:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078078.952728:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078078.952731:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078078.952740:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616847616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078078.952746:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078078.952748:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078078.952749:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078078.952751:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5064 00000100:00080000:1.0:1769078078.952763:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078078.952766:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078078.952769:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769078078.952778:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078078.952779:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616847744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078078.952787:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078078.952789:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078078.952791:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078078.952794:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5064 00000100:00080000:1.0:1769078078.952796:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078078.952798:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078078.952802:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078078.952808:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616847872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078078.952812:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078078.952814:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078078.952816:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078078.952819:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5064 00000100:00080000:1.0:1769078078.952821:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078078.952824:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078078.952826:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078078.952832:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078078/real 1769078078] req@ffff995b868c5c00 x1855007616847360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078078.952835:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616848000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078078.952847:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078078.952850:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078078.952851:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078078.952854:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5064 00000100:00080000:1.0:1769078078.952857:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078078.952858:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616847360/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078078.952861:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078078.952867:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078078.952877:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616848128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078078.952879:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078078.952890:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078078.952896:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616847360:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078078.952925:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616847488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078078.952949:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078078.952962:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078078/real 1769078078] req@ffff995b868c5500 x1855007616847488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078078.952974:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616847488/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078078.952982:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078078.952985:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078078.952989:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616847488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078078.952994:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616847616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078078.953012:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078078.953022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078078/real 1769078078] req@ffff995b868c4700 x1855007616847616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078078.953046:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616847616/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078078.953056:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078078.953059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078078.953062:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616847616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078078.953068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616847744:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078078.953087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078078.953097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078078/real 1769078078] req@ffff995b868c7800 x1855007616847744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078078.953108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616847744/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078078.953118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078078.953121:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078078.953125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616847744:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078078.953130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616847872:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078078.953149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078078.953175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078078/real 1769078078] req@ffff995b868c7480 x1855007616847872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078078.953187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616847872/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078078.953196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078078.953199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078078.953203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616847872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078078.953208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616848000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078078.953227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078078.953236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078078/real 1769078078] req@ffff995b868c7100 x1855007616848000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078078.953249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616848000/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078078.953256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078078.953259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078078.953263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616848000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078078.953269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616848128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078078.953301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078078.953312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078078/real 1769078078] req@ffff995b868c6a00 x1855007616848128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078078.953325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616848128/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078078.953332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078078.953335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078078.953339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616848128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078079.656579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078079.656645:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078079.976513:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078079.976526:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078079.976559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078079.976559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078080.680517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078080.680582:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078081.000589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078081.000606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078081.000607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078081.000616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078081.704577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078081.704701:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078082.024581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078082.024581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078082.024597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078082.024597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078082.728589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078082.728651:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078083.048577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078083.048578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078083.048595:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078083.048595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078083.752589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078083.752664:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078084.072561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078084.072566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078084.072582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078084.072582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078084.072622:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078084.072628:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078084.072633:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078084.072645:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5069 00000100:00080000:3.0:1769078084.072656:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078084.072663:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078084.072702:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616848256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078084.072724:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078084.072726:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078084.072728:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078084.072730:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5069 00000100:00080000:3.0:1769078084.072733:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078084.072737:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078084.072745:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078084.072755:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616848384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078084.072757:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616848256:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078084.072770:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078084.072772:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078084.072773:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078084.072776:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5069 00000100:00080000:3.0:1769078084.072777:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078084.072779:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078084.072782:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078084.072790:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616848512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078084.072795:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078084.072797:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078084.072798:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078084.072801:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5069 00000100:00080000:3.0:1769078084.072803:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078084.072805:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078084.072808:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078084.072814:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616848640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078084.072819:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078084.072821:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078084.072822:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078084.072824:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5069 00000100:00080000:3.0:1769078084.072826:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078084.072830:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078084.072831:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078084.072836:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078084.072843:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616848768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078084.072849:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078084.072851:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078084.072853:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078084.072856:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5069 00000100:00080000:3.0:1769078084.072858:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078084.072860:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078084.072862:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078084.072869:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616848896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078084.072875:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078084.072879:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078084/real 1769078084] req@ffff995b84559880 x1855007616848256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078084.072879:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078084.072880:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078084.072886:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5069 00000100:00080000:3.0:1769078084.072889:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078084.072894:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078084.072899:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078084.072908:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616849024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078084.072923:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616848256/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078084.072935:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078084.072940:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078084.072944:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616848256:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078084.072966:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616848384:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078084.072987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078084.072998:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078084/real 1769078084] req@ffff995b84558700 x1855007616848384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078084.073008:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616848384/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078084.073017:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078084.073020:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078084.073023:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616848384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078084.073029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616848512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078084.073047:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078084.073058:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078084/real 1769078084] req@ffff995b8455b480 x1855007616848512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078084.073098:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616848512/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078084.073113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078084.073116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078084.073119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616848512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078084.073125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616848640:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078084.073144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078084.073154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078084/real 1769078084] req@ffff995b8455ad80 x1855007616848640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078084.073162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616848640/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078084.073167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078084.073169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078084.073171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616848640:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078084.073174:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616848768:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078084.073185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078084.073190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078084/real 1769078084] req@ffff995b8455a300 x1855007616848768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078084.073197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616848768/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078084.073202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078084.073203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078084.073205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616848768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078084.073209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616848896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078084.073217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078084.073222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078084/real 1769078084] req@ffff995b84559c00 x1855007616848896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078084.073228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616848896/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078084.073233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078084.073234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078084.073236:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616848896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078084.073238:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616849024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078084.073246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078084.073251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078084/real 1769078084] req@ffff995b84558000 x1855007616849024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078084.073256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616849024/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078084.073261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078084.073262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078084.073264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616849024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078084.776549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078084.776607:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078085.096543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078085.096543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078085.096544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078085.096548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078085.800563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078085.800625:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078086.120553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078086.120553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078086.120553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078086.120554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078086.824549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078086.824618:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078087.144560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078087.144561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078087.144569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078087.144569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078087.848558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078087.848621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078088.168511:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078088.168521:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078088.168540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078088.168541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078088.872543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078088.872621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078089.192543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078089.192543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078089.192543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078089.192581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078089.192613:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078089.192619:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078089.192624:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078089.192634:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5074 00000100:00080000:2.0:1769078089.192645:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078089.192653:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078089.192694:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616849152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078089.192718:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078089.192721:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078089.192723:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078089.192727:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5074 00000100:00080000:2.0:1769078089.192731:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078089.192735:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078089.192745:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078089.192750:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616849152:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078089.192759:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616849280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078089.192771:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078089.192774:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078089.192776:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078089.192780:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5074 00000100:00080000:2.0:1769078089.192784:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078089.192787:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078089.192792:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078089.192802:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616849408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078089.192812:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078089.192812:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078089.192818:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078089.192820:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078089.192825:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5074 00000100:00080000:2.0:1769078089.192829:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078089.192833:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078089.192837:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078089.192846:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616849536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078089.192854:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078089.192855:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078089/real 1769078089] req@ffff995b85ac9c00 x1855007616849152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078089.192864:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078089.192866:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078089.192871:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5074 00000100:00100000:0.0:1769078089.192872:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616849152/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078089.192874:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078089.192880:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078089.192885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769078089.192885:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769078089.192892:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078089.192898:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616849152:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078089.192898:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616849664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078089.192910:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078089.192913:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078089.192914:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078089.192917:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5074 00000100:00080000:2.0:1769078089.192919:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078089.192925:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078089.192928:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078089.192929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616849280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078089.192936:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616849792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078089.192943:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078089.192945:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078089.192946:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078089.192948:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5074 00000100:00080000:2.0:1769078089.192951:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078089.192954:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078089.192957:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769078089.192958:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078089.192967:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616849920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078089.192994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078089/real 1769078089] req@ffff995b85acb100 x1855007616849280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078089.193005:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616849280/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078089.193011:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078089.193014:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078089.193018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616849280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078089.193024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616849408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078089.193043:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078089.193053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078089/real 1769078089] req@ffff995b85aca300 x1855007616849408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078089.193064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616849408/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078089.193071:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078089.193073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078089.193075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616849408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078089.193079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616849536:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078089.193092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078089.193098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078089/real 1769078089] req@ffff995b85acb800 x1855007616849536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078089.193105:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616849536/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078089.193110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078089.193111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078089.193113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616849536:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078089.193116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616849664:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078089.193125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078089.193131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078089/real 1769078089] req@ffff995b85ac8a80 x1855007616849664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078089.193137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616849664/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078089.193142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078089.193144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078089.193146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616849664:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078089.193158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616849792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078089.193167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078089.193172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078089/real 1769078089] req@ffff995b85ac8380 x1855007616849792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078089.193179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616849792/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078089.193183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078089.193185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078089.193186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616849792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078089.193189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616849920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078089.193197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078089.193202:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078089/real 1769078089] req@ffff995b85ac9880 x1855007616849920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078089.193208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616849920/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078089.193213:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078089.193215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078089.193216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616849920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078089.896553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078089.896629:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078090.216550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078090.216550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078090.216550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078090.216694:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078090.920517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078090.920561:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078091.240555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078091.240555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078091.240556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078091.240580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078091.944535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078091.944586:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078092.264549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078092.264549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078092.264549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078092.264552:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078092.968554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078092.968615:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078093.288528:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078093.288557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078093.288760:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078093.288762:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078093.992631:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078093.992648:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078094.312574:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078094.312577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078094.312585:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078094.312591:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078094.312605:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5080 00000100:00100000:0.0:1769078094.312606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078094.312613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078094.312621:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078094.312634:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078094.312640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078094.312683:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616850048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078094.312714:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078094.312717:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078094.312719:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078094.312723:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5080 00000100:00080000:3.0:1769078094.312727:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078094.312732:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078094.312742:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078094.312755:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616850048:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078094.312758:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616850176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078094.312775:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078094.312777:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078094.312779:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078094.312783:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5080 00000100:00080000:3.0:1769078094.312786:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078094.312789:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078094.312793:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078094.312802:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616850304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078094.312809:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078094.312812:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078094.312813:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078094.312815:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5080 00000100:00080000:3.0:1769078094.312820:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078094.312823:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078094.312827:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769078094.312833:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078094.312839:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616850432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078094.312849:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078094.312853:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078094.312854:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078094.312857:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5080 00000100:00080000:3.0:1769078094.312860:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078094.312864:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078094.312869:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078094.312878:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078094/real 1769078094] req@ffff995b84559500 x1855007616850048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078094.312881:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616850560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078094.312948:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078094.312952:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078094.312955:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078094.312961:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5080 00000100:00100000:0.0:1769078094.312964:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616850048/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078094.312966:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078094.312981:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078094.312985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769078094.312988:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078094.312992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078094.313001:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616850048:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078094.313009:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616850688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078094.313021:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078094.313024:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078094.313026:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078094.313028:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616850176:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078094.313034:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5080 00000100:00080000:3.0:1769078094.313038:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078094.313041:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078094.313046:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769078094.313058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078094.313059:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616850816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078094.313074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078094/real 1769078094] req@ffff995b8455bb80 x1855007616850176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078094.313087:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616850176/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078094.313094:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078094.313097:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078094.313100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616850176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078094.313104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616850304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078094.313116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078094.313122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078094/real 1769078094] req@ffff995b84558a80 x1855007616850304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078094.313130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616850304/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078094.313137:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078094.313139:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078094.313142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616850304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078094.313148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616850432:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078094.313166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078094.313176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078094/real 1769078094] req@ffff995b8455a680 x1855007616850432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078094.313189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616850432/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078094.313206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078094.313210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078094.313214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616850432:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078094.313221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616850560:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078094.313238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078094.313246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078094/real 1769078094] req@ffff995b8455b100 x1855007616850560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078094.313254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616850560/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078094.313260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078094.313263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078094.313265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616850560:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078094.313269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616850688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078094.313280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078094.313285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078094/real 1769078094] req@ffff995b8455aa00 x1855007616850688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078094.313293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616850688/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078094.313298:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078094.313300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078094.313303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616850688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078094.313318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616850816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078094.313328:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078094.313335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078094/real 1769078094] req@ffff995b84559500 x1855007616850816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078094.313342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616850816/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078094.313348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078094.313350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078094.313352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616850816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078095.016603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078095.016658:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078095.336541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078095.336595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078095.336609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078095.336630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078096.040584:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078096.040670:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078096.360603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078096.360606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078096.360616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078096.360667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078097.064598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078097.064716:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078097.384563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078097.384575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078097.384578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078097.384630:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078098.088557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078098.088643:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078098.408559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078098.408561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078098.408571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078098.408612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078099.112581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078099.112622:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078099.432578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078099.432580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769078099.432616:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078099.432625:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078099.432629:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078099.432631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078099.432644:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5085 00000100:00080000:2.0:1769078099.432658:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078099.432667:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078099.432676:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078099.432708:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616850944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078099.432734:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078099.432738:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078099.432739:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078099.432743:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5085 00000100:00080000:2.0:1769078099.432746:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078099.432751:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078099.432764:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078099.432774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616850944:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078099.432783:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616851072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078099.432800:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078099.432818:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078099.432820:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078099.432824:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5085 00000100:00080000:2.0:1769078099.432828:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078099.432831:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078099.432855:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078099.432872:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616851200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078099.432879:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078099.432882:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078099.432883:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078099.432896:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5085 00000800:00000100:0.0:1769078099.432902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078099.432905:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078099.432912:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078099.432917:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078099.432929:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616851328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078099.432936:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078099.432939:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078099.432941:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078099.432944:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5085 00000100:00080000:2.0:1769078099.432957:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078099.432960:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078099.432964:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078099.432971:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078099/real 1769078099] req@ffff995b85acbb80 x1855007616850944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078099.432977:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616851456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078099.433013:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078099.433016:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078099.433018:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078099.433022:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5085 00000100:00080000:2.0:1769078099.433027:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078099.433034:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078099.433035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616850944/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769078099.433041:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078099.433065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769078099.433065:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616851584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078099.433076:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078099.433088:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616850944:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078099.433103:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078099.433107:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078099.433109:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078099.433115:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5085 00000100:00080000:2.0:1769078099.433119:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078099.433123:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078099.433127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851072:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769078099.433129:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078099.433150:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616851712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078099.433166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078099.433179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078099/real 1769078099] req@ffff995b85acaa00 x1855007616851072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078099.433194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616851072/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078099.433200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078099.433202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078099.433204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078099.433209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078099.433220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078099.433227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078099/real 1769078099] req@ffff995b85ac9880 x1855007616851200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078099.433234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616851200/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078099.433241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078099.433243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078099.433245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078099.433250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851328:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078099.433260:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078099.433266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078099/real 1769078099] req@ffff995b85ac8380 x1855007616851328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078099.433273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616851328/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078099.433280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078099.433283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078099.433286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851328:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078099.433292:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851456:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078099.433310:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078099.433322:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078099/real 1769078099] req@ffff995b85ac8a80 x1855007616851456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078099.433335:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616851456/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078099.433345:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078099.433349:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078099.433352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078099.433357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616851584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078099.433374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078099.433382:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078099/real 1769078099] req@ffff995b85acb800 x1855007616851584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078099.433392:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616851584/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078099.433401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078099.433404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078099.433408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616851584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078099.433431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616851712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078099.433454:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078099.433470:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078099/real 1769078099] req@ffff995b85aca300 x1855007616851712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078099.433482:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616851712/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078099.433505:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078099.433508:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078099.433510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616851712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078100.136545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078100.136618:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078100.456592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078100.456602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078100.456638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078100.456652:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078101.160552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078101.160612:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078101.480578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078101.480602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078101.480611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078101.480623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078102.184537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078102.184641:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078102.504566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078102.504566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078102.504596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078102.504622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078103.208537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078103.208607:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078103.528570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078103.528570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078103.528685:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078103.528781:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078104.232563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078104.232650:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078104.552576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078104.552576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078104.552624:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078104.552672:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078104.552682:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078104.552688:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078104.552699:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5090 00000100:00080000:1.0:1769078104.552710:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078104.552718:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078104.552767:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616851840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078104.552794:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078104.552798:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078104.552801:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078104.552805:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5090 00000100:00080000:1.0:1769078104.552809:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078104.552814:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078104.552825:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078104.552838:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616851968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078104.552848:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078104.552851:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078104.552853:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078104.552857:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5090 00000100:00080000:1.0:1769078104.552859:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078104.552862:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078104.552867:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078104.552878:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616852096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078104.552886:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078104.552889:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078104.552891:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078104.552894:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5090 00000100:00080000:1.0:1769078104.552896:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078104.552899:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078104.552904:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078104.552913:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616852224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078104.552921:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078104.552957:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078104.552969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616851840:192.168.204.132@tcp:250:kworker.0 00000800:00000100:3.0:1769078104.553098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078104.553129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078104/real 1769078104] req@ffff995b868c6300 x1855007616851840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078104.553141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616851840/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078104.553153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078104.553157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078104.553160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616851840:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078104.553165:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078104.553290:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078104.553301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078104/real 1769078104] req@ffff995b868c6d80 x1855007616851968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078104.553356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616851968/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078104.553367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078104.553372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078104.553376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616851968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078104.553384:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078104.553405:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078104.553414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078104/real 1769078104] req@ffff995b868c4a80 x1855007616852096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078104.553422:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616852096/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078104.553430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078104.553432:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078104.553435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078104.553439:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852224:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078104.553450:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078104.553482:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078104/real 1769078104] req@ffff995b868c6a00 x1855007616852224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078104.553492:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616852224/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078104.553499:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078104.553501:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078104.553504:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852224:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769078104.553567:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078104.553569:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078104.553571:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5090 00000100:00080000:1.0:1769078104.553575:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078104.553577:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078104.553581:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078104.553591:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616852352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078104.553606:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078104.553609:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078104.553611:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078104.553614:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5090 00000100:00080000:1.0:1769078104.553617:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078104.553619:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078104.553622:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078104.553629:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616852480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078104.553635:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078104.553637:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078104.553638:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078104.553640:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5090 00000100:00080000:1.0:1769078104.553642:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078104.553644:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078104.553646:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078104.553651:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616852608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078104.553698:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852352:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078104.553715:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078104.553739:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078104/real 1769078104] req@ffff995b868c7100 x1855007616852352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078104.553748:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616852352/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078104.553754:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078104.553756:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078104.553758:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078104.553763:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616852480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078104.553776:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078104.553783:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078104/real 1769078104] req@ffff995b868c7480 x1855007616852480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078104.553791:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616852480/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078104.553796:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078104.553798:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078104.553800:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616852480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078104.553803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616852608:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078104.553813:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078104.553819:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078104/real 1769078104] req@ffff995b868c7800 x1855007616852608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078104.553826:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616852608/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078104.553831:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078104.553833:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078104.553835:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616852608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078105.256573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078105.256640:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078105.576578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078105.576579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078105.576618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078105.576638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769078106.280662:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078106.280744:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078106.600596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078106.600600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078106.600607:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078106.600656:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078107.304555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078107.304660:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078107.624604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078107.624604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078107.624606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078107.624695:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078108.328553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078108.328611:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078108.648604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078108.648604:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078108.648608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078108.648623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078109.352570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078109.352663:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078109.672651:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078109.672651:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078109.672652:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078109.672669:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078109.672708:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078109.672716:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078109.672721:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078109.672733:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5095 00000100:00080000:0.0:1769078109.672750:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078109.672759:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078109.672801:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616852736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078109.672833:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078109.672837:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078109.672839:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078109.672844:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5095 00000100:00080000:0.0:1769078109.672848:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078109.672852:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078109.672863:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078109.672879:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616852864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078109.672880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616852736:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078109.672906:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078109.672910:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078109.672912:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078109.672916:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5095 00000100:00080000:0.0:1769078109.672920:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078109.672924:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078109.672929:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078109.672942:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616852992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078109.672963:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078109.672966:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078109.672968:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769078109.672975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078109.672976:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5095 00000100:00080000:0.0:1769078109.672982:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078109.672987:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078109.672993:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078109.673007:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616853120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078109.673018:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078109.673022:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078109.673024:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078109.673028:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5095 00000100:00080000:0.0:1769078109.673032:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078109.673036:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078109.673042:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078109.673050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078109/real 1769078109] req@ffff995b84641f80 x1855007616852736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078109.673055:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616853248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078109.673081:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078109.673085:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078109.673087:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078109.673093:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5095 00000100:00100000:3.0:1769078109.673094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616852736/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078109.673097:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078109.673104:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078109.673113:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769078109.673116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078109.673126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078109.673127:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616853376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078109.673135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616852736:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078109.673144:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078109.673149:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078109.673151:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078109.673156:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5095 00000100:00080000:0.0:1769078109.673160:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078109.673163:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078109.673168:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078109.673171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078109.673181:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616853504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078109.673203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078109.673222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078109/real 1769078109] req@ffff995b84640a80 x1855007616852864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078109.673238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616852864/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078109.673248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078109.673252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078109.673256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078109.673262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078109.673286:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078109.673300:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078109/real 1769078109] req@ffff995b84641c00 x1855007616852992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078109.673314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616852992/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078109.673333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078109.673337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078109.673341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616852992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078109.673348:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616853120:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078109.673372:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078109.673384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078109/real 1769078109] req@ffff995b84640380 x1855007616853120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078109.673399:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616853120/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078109.673409:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078109.673413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078109.673417:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616853120:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078109.673423:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616853248:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078109.673444:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078109.673482:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078109/real 1769078109] req@ffff995b84640000 x1855007616853248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078109.673497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616853248/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078109.673508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078109.673511:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078109.673516:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616853248:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078109.673524:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616853376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078109.673547:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078109.673560:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078109/real 1769078109] req@ffff995b84642680 x1855007616853376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078109.673574:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616853376/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078109.673583:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078109.673587:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078109.673591:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616853376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078109.673614:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616853504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078109.673653:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078109.673667:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078109/real 1769078109] req@ffff995b84640e00 x1855007616853504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078109.673682:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616853504/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078109.673693:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078109.673696:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078109.673700:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616853504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078110.376593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078110.376679:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078110.696586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078110.696590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078110.696631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078110.696656:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078111.400579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078111.400672:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078111.720571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078111.720571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078111.720641:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078111.720655:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078112.424571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078112.424638:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078112.744547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078112.744551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078112.744577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078112.744610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078113.448581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078113.448629:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078113.768580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078113.768582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078113.768592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078113.768633:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078114.472578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078114.472682:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078114.792580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078114.792626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078114.792662:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078114.792667:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078114.792671:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078114.792680:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078114.792685:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078114.792737:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5100 00000100:00080000:3.0:1769078114.792750:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078114.792757:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078114.792830:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616853632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078114.792872:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078114.792877:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078114.792879:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078114.792884:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5100 00000100:00080000:3.0:1769078114.792889:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078114.792894:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078114.792906:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078114.792920:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616853760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078114.792931:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078114.792935:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078114.792937:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078114.792940:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5100 00000100:00080000:3.0:1769078114.792944:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078114.792948:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078114.792953:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078114.792965:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616853888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078114.792975:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078114.792977:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078114.792979:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078114.792983:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5100 00000100:00080000:3.0:1769078114.792987:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078114.792991:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078114.792996:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078114.793011:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616854016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078114.793020:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078114.793023:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078114.793025:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078114.793028:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5100 00000100:00080000:3.0:1769078114.793032:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078114.793035:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078114.793039:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078114.793047:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616854144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078114.793055:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078114.793058:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078114.793060:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078114.793064:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5100 00000100:00080000:3.0:1769078114.793068:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078114.793072:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078114.793077:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078114.793087:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616854272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078114.793096:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078114.793099:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078114.793101:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078114.793105:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5100 00000100:00080000:3.0:1769078114.793108:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078114.793111:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078114.793116:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078114.793126:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616854400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078114.793392:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616853632:192.168.204.132@tcp:250:kworker.0 00000800:00000100:0.0:1769078114.793509:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078114.793581:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078114/real 1769078114] req@ffff995b84558000 x1855007616853632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078114.793604:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616853632/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078114.793617:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078114.793622:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078114.793627:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616853632:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078114.793637:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616853760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078114.793665:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078114.793681:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078114/real 1769078114] req@ffff995b84559c00 x1855007616853760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078114.793695:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616853760/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078114.793704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078114.793709:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078114.793712:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616853760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078114.793720:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616853888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078114.793744:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078114.793755:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078114/real 1769078114] req@ffff995b8455a300 x1855007616853888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078114.793764:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616853888/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078114.793772:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078114.793775:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078114.793777:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616853888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078114.793782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854016:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078114.793796:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078114.793802:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078114/real 1769078114] req@ffff995b8455ad80 x1855007616854016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078114.793814:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616854016/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078114.793824:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078114.793828:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078114.793831:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854016:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078114.793837:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854144:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078114.793860:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078114.793872:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078114/real 1769078114] req@ffff995b8455b480 x1855007616854144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078114.793886:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616854144/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078114.793896:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078114.793898:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078114.793901:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854144:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078114.793906:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616854272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078114.793923:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078114.793930:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078114/real 1769078114] req@ffff995b84558700 x1855007616854272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078114.793938:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616854272/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078114.793944:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078114.793947:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078114.793949:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616854272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078114.793953:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616854400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078114.793963:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078114.793969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078114/real 1769078114] req@ffff995b84559880 x1855007616854400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078114.793982:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616854400/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078114.793988:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078114.793990:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078114.793992:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616854400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078115.496565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078115.496620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078115.816567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078115.816570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078115.816599:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078115.816600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078116.520559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078116.520613:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078116.840526:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078116.840544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078116.840553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078116.840558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078117.544552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078117.544602:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078117.864558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078117.864562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078117.864575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078117.864576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078118.568557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078118.568614:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078118.888568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078118.888568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078118.888569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078118.888578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078119.592531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078119.592579:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078119.912575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078119.912575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078119.912576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078119.912708:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078119.912708:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078119.912714:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078119.912718:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078119.912726:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5105 00000100:00080000:2.0:1769078119.912736:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078119.912741:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078119.912775:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616854528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078119.912795:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078119.912798:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078119.912799:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078119.912802:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5105 00000100:00080000:2.0:1769078119.912804:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078119.912806:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078119.912819:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078119.912828:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616854656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078119.912835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616854528:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078119.912838:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078119.912841:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078119.912842:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078119.912845:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5105 00000100:00080000:2.0:1769078119.912848:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078119.912850:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078119.912853:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078119.912861:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616854784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078119.912868:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078119.912870:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078119.912871:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078119.912873:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5105 00000100:00080000:2.0:1769078119.912876:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078119.912879:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078119.912881:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078119.912889:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616854912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078119.912894:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078119.912895:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078119.912899:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078119.912901:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078119.912904:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5105 00000100:00080000:2.0:1769078119.912907:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078119.912910:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078119.912912:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078119.912919:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616855040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078119.912928:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078119.912932:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078119/real 1769078119] req@ffff995b85acb100 x1855007616854528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078119.912933:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078119.912937:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078119.912946:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5105 00000100:00080000:2.0:1769078119.912951:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078119.912952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616854528/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078119.912959:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078119.912962:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769078119.912966:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078119.912967:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078119.912972:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616854528:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078119.912983:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616855168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078119.912995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854656:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078119.912996:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078119.913000:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078119.913002:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078119.913005:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5105 00000100:00080000:2.0:1769078119.913009:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078119.913012:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:0.0:1769078119.913015:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:2.0:1769078119.913019:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078119.913027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078119/real 1769078119] req@ffff995b85ac9c00 x1855007616854656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078119.913035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616854656/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078119.913041:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616855296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078119.913042:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078119.913049:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078119.913054:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078119.913061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078119.913073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078119.913082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078119/real 1769078119] req@ffff995b85aca300 x1855007616854784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078119.913093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616854784/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078119.913103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078119.913106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078119.913110:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078119.913116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854912:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078119.913133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078119.913149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078119/real 1769078119] req@ffff995b85acb800 x1855007616854912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078119.913157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616854912/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078119.913163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078119.913165:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078119.913167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616854912:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078119.913170:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855040:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078119.913181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078119.913186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078119/real 1769078119] req@ffff995b85ac8a80 x1855007616855040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078119.913193:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616855040/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078119.913198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078119.913199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078119.913201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078119.913217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616855168:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078119.913227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078119.913233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078119/real 1769078119] req@ffff995b85ac8380 x1855007616855168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078119.913239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616855168/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078119.913248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078119.913250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078119.913252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616855168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078119.913255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616855296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078119.913263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078119.913267:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078119/real 1769078119] req@ffff995b85ac9880 x1855007616855296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078119.913273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616855296/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078119.913278:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078119.913280:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078119.913281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616855296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078120.616554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078120.616606:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078120.936522:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078120.936558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078120.936575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078120.936581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078121.640527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078121.640564:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078121.960543:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078121.960543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078121.960543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078121.960577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078122.664548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078122.664630:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078122.984554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078122.984554:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078122.984555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078122.984555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078123.688572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078123.688630:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078124.008546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078124.008559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078124.008581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078124.008589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078124.712566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078124.712653:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078125.032607:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078125.032608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078125.032609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078125.032614:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078125.032625:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078125.032633:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078125.032649:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5110 00000100:00080000:2.0:1769078125.032662:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078125.032668:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078125.032671:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078125.032720:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616855424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078125.032752:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078125.032756:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078125.032759:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078125.032764:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5110 00000100:00080000:2.0:1769078125.032770:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078125.032776:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078125.032786:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078125.032791:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616855424:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078125.032810:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616855552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078125.032825:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078125.032828:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078125.032832:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078125.032837:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5110 00000100:00080000:2.0:1769078125.032841:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078125.032846:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078125.032853:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078125.032870:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616855680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078125.032877:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078125.032891:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078125.032896:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078125.032899:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078125.032906:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5110 00000100:00080000:2.0:1769078125.032911:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078125.032916:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078125.032923:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078125.032928:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078125/real 1769078125] req@ffff995b85acaa00 x1855007616855424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078125.032941:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616855808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078125.032947:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616855424/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078125.032962:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078125.032974:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078125.032974:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078125.032978:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078125.032982:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769078125.032989:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5110 00000100:00100000:0.0:1769078125.032991:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616855424:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078125.033000:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078125.033009:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078125.033016:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078125.033029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078125.033037:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616855936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078125.033052:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078125.033053:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078125.033059:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078125.033062:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078125.033067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078125/real 1769078125] req@ffff995b85acbb80 x1855007616855552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078125.033072:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5110 00000100:00100000:0.0:1769078125.033087:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616855552/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078125.033087:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078125.033099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078125.033100:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078125.033104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078125.033112:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855552:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769078125.033115:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078125.033118:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078125.033143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078125.033144:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616856064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078125.033153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078125/real 1769078125] req@ffff995b85ac9880 x1855007616855680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078125.033165:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078125.033185:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078125.033191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616855680/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078125.033192:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078125.033207:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5110 00000100:00080000:2.0:1769078125.033218:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078125.033229:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078125.033230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078125.033246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:2.0:1769078125.033247:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078125.033255:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078125.033269:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616856192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078125.033270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855808:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078125.033304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078125.033315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078125/real 1769078125] req@ffff995b85ac8380 x1855007616855808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078125.033324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616855808/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078125.033332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078125.033335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078125.033337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855808:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078125.033351:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855936:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078125.033361:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078125.033367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078125/real 1769078125] req@ffff995b85ac8a80 x1855007616855936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078125.033397:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616855936/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078125.033405:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078125.033407:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078125.033410:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616855936:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078125.033415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616856064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078125.033430:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078125.033438:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078125/real 1769078125] req@ffff995b85acb800 x1855007616856064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078125.033446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616856064/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078125.033473:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078125.033476:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078125.033478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616856064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078125.033483:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616856192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078125.033495:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078125.033503:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078125/real 1769078125] req@ffff995b85aca300 x1855007616856192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078125.033511:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616856192/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078125.033517:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078125.033520:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078125.033522:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616856192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078125.736527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078125.736621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078126.056569:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078126.056569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078126.056573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078126.056606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078126.760565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078126.760632:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078127.080574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078127.080574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078127.080574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078127.080625:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078127.784556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078127.784633:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078128.104548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078128.104557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078128.104602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078128.104631:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078128.808568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078128.808627:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078129.128647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078129.128648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078129.128648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078129.128651:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078129.832561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078129.832655:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078130.152582:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078130.152589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078130.152589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078130.152602:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078130.152609:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078130.152621:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5115 00000100:00100000:3.0:1769078130.152627:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078130.152630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769078130.152633:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078130.152642:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078130.152684:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616856320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078130.152708:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078130.152713:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078130.152714:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078130.152717:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5115 00000100:00080000:2.0:1769078130.152720:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078130.152726:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078130.152736:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078130.152748:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616856320:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078130.152749:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616856448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078130.152772:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078130.152774:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078130.152775:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078130.152780:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5115 00000100:00080000:2.0:1769078130.152783:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078130.152786:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078130.152789:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078130.152800:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616856576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078130.152806:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078130.152809:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078130.152810:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078130.152813:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5115 00000100:00080000:2.0:1769078130.152819:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078130.152822:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078130.152827:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078130.152840:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616856704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078130.152851:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078130.152855:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078130.152857:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078130.152860:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5115 00000100:00080000:2.0:1769078130.152865:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078130.152881:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078130.152888:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078130.152903:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616856832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078130.152913:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078130.152918:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078130.152919:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078130.152926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616856448:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078130.152927:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5115 00000100:00080000:2.0:1769078130.152936:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078130.152944:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078130.152950:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078130.152963:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616856960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078130.152977:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078130.152979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616856576:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078130.152989:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078130.153000:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078130.153007:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5115 00000100:00080000:2.0:1769078130.153013:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078130.153017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616856704:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769078130.153023:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078130.153033:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078130.153054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616856832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078130.153062:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616857088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078130.153096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616856960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078130.153122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616857088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078130.856547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078130.856656:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078131.176594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078131.176594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078131.176633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078131.176679:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078131.880563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078131.880635:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078132.200555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078132.200557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078132.200558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078132.200607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078132.904529:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078132.904583:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078133.224556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078133.224556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078133.224606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078133.224715:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:3.0:1769078133.224738:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:3.0:1769078133.224790:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769078133.224812:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769078133.224855:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769078133.224867:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769078133.224877:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769078133.224885:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769078133.224893:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:3.0:1769078133.224901:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769078133.225138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078130/real 1769078133] req@ffff995b85ac8380 x1855007616856960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078133.225164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616856960/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078133.225177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078133.225184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078133.225189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616856960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078133.225198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078130/real 1769078133] req@ffff995b85ac9880 x1855007616857088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078133.225212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616857088/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078133.225221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078133.225226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078133.225230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616857088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078133.225267:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078130/real 1769078133] req@ffff995b85acb100 x1855007616856448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078133.225282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616856448/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078133.225292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078133.225296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078133.225299:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616856448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078133.225308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078130/real 1769078133] req@ffff995b85aca300 x1855007616856576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078133.225321:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616856576/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078133.225339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078133.225344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078133.225348:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616856576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078133.225355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078130/real 1769078133] req@ffff995b85acb800 x1855007616856704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078133.225370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616856704/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078133.225379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078133.225383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078133.225387:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616856704:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078133.225394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078130/real 1769078133] req@ffff995b85ac8a80 x1855007616856832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078133.225408:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616856832/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078133.225418:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078133.225421:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078133.225424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616856832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078133.225432:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078130/real 1769078133] req@ffff995b85ac9c00 x1855007616856320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078133.225447:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616856320/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078133.225476:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078133.225480:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078133.225484:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616856320:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078133.928512:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078133.928644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078134.248571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078134.248582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078134.248617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078134.248631:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078134.952538:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078134.952612:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078135.272617:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078135.272622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078135.272622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078135.272659:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078135.272666:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078135.272669:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078135.272678:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5120 00000100:00080000:2.0:1769078135.272689:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078135.272696:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078135.272703:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078135.272736:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616857216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078135.272770:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078135.272777:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078135.272779:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078135.272783:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5120 00000100:00080000:2.0:1769078135.272788:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078135.272794:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078135.272805:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078135.272817:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616857216:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078135.272824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616857344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078135.272837:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078135.272841:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078135.272843:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078135.272847:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5120 00000100:00080000:2.0:1769078135.272851:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078135.272855:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078135.272860:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078135.272875:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616857472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078135.272884:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078135.272889:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078135.272891:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078135.272895:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078135.272900:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5120 00000100:00080000:2.0:1769078135.272905:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078135.272909:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078135.272915:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078135.272927:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616857600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078135.272933:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078135/real 1769078135] req@ffff995b85acbb80 x1855007616857216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078135.272955:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078135.272958:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078135.272960:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078135.272965:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616857216/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078135.272966:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5120 00000100:00080000:2.0:1769078135.272973:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078135.272981:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078135.272988:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769078135.272989:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769078135.272998:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078135.273006:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616857216:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078135.273008:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616857728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078135.273021:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078135.273024:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078135.273026:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078135.273029:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5120 00000100:00080000:2.0:1769078135.273032:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078135.273035:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078135.273039:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078135.273041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616857344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078135.273050:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616857856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078135.273059:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078135.273062:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078135.273064:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769078135.273070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078135.273070:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5120 00000100:00080000:2.0:1769078135.273077:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078135.273081:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078135.273085:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078135.273089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078135/real 1769078135] req@ffff995b85acaa00 x1855007616857344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078135.273096:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616857984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078135.273107:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616857344/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078135.273114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078135.273119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078135.273123:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616857344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078135.273128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616857472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078135.273140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078135.273147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078135/real 1769078135] req@ffff995b85ac9c00 x1855007616857472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078135.273156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616857472/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078135.273164:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078135.273167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078135.273169:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616857472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078135.273175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616857600:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078135.273197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078135.273222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078135/real 1769078135] req@ffff995b85ac8a80 x1855007616857600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078135.273232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616857600/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078135.273239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078135.273242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078135.273245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616857600:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078135.273250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616857728:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078135.273266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078135.273283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078135/real 1769078135] req@ffff995b85acb800 x1855007616857728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078135.273303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616857728/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078135.273310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078135.273312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078135.273314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616857728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078135.273337:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616857856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078135.273357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078135.273369:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078135/real 1769078135] req@ffff995b85aca300 x1855007616857856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078135.273382:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616857856/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078135.273389:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078135.273393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078135.273397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616857856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078135.273403:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616857984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078135.273423:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078135.273435:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078135/real 1769078135] req@ffff995b85acb100 x1855007616857984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078135.273487:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616857984/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078135.273498:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078135.273502:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078135.273506:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616857984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078135.976563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078135.976631:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078136.296543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078136.296558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078136.296564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078136.296587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078137.000542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078137.000614:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078137.320562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078137.320566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078137.320605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078137.320606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078138.024560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078138.024613:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078138.344519:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078138.344553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078138.344565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078138.344575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078139.048520:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078139.048588:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078139.368551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078139.368552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078139.368582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078139.368597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078140.072551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078140.072653:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078140.392598:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078140.392604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078140.392619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078140.392658:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078140.392666:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078140.392675:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078140.392682:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078140.392693:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5126 00000100:00080000:3.0:1769078140.392707:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078140.392717:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078140.392760:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616858112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078140.392802:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078140.392807:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078140.392812:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078140.392821:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5126 00000100:00080000:3.0:1769078140.392829:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078140.392836:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078140.392841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616858112:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769078140.392852:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078140.392879:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616858240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078140.392895:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078140.392899:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078140.392902:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078140.392928:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078140.392989:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5126 00000100:00100000:1.0:1769078140.392996:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078140/real 1769078140] req@ffff995b84559880 x1855007616858112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078140.392999:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078140.393011:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078140.393020:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078140.393035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616858112/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078140.393048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078140.393052:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078140.393056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616858112:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078140.393080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616858240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078140.393102:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078140.393112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078140/real 1769078140] req@ffff995b84558700 x1855007616858240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078140.393141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616858240/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078140.393147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078140.393151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078140.393156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616858240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078140.393168:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616858368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078140.393198:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078140.393203:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769078140.393220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616858368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078140.393242:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078140.393247:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078140.393260:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5126 00000100:00100000:1.0:1769078140.393263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078140/real 1769078140] req@ffff995b8455b480 x1855007616858368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078140.393271:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078140.393283:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078140.393284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616858368/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078140.393301:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769078140.393304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078140.393311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078140.393314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616858368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078140.393326:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616858496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078140.393355:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078140.393360:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078140.393362:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078140.393367:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5126 00000100:00080000:3.0:1769078140.393372:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078140.393378:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078140.393385:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078140.393386:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616858496:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078140.393410:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616858624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078140.393416:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078140.393428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078140/real 1769078140] req@ffff995b8455ad80 x1855007616858496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078140.393432:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078140.393442:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769078140.393443:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616858496/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078140.393446:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078140.393492:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5126 00000100:00080000:1.0:1769078140.393493:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078140.393508:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769078140.393509:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078140.393520:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616858496:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769078140.393525:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078140.393535:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078140.393543:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616858624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078140.393554:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616858752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078140.393565:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078140.393584:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078140/real 1769078140] req@ffff995b8455a300 x1855007616858624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078140.393584:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078140.393606:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078140.393611:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078140.393613:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616858624/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078140.393624:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5126 00000100:00080000:1.0:1769078140.393631:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078140.393635:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078140.393636:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078140.393641:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616858624:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769078140.393649:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078140.393656:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616858752:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769078140.393660:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078140.393672:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078140.393685:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616858880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078140.393688:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078140/real 1769078140] req@ffff995b84559c00 x1855007616858752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078140.393751:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616858752/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078140.393765:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078140.393770:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078140.393775:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616858752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078140.393790:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616858880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078140.393813:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078140.393822:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078140/real 1769078140] req@ffff995b84558000 x1855007616858880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078140.393830:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616858880/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078140.393836:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078140.393838:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078140.393841:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616858880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078141.096514:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078141.096619:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078141.416600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078141.416606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078141.416643:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078141.416657:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078142.120563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078142.120630:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078142.440582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078142.440585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078142.440600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078142.440647:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078143.144566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078143.144623:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078143.464547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078143.464573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078143.464611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078143.464648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078144.168557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078144.168616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078144.488506:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078144.488541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078144.488545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078144.488576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078145.192561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078145.192620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078145.512540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078145.512545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769078145.512587:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078145.512597:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078145.512601:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078145.512611:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5131 00000100:00080000:1.0:1769078145.512623:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078145.512629:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078145.512667:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616859008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078145.512697:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078145.512702:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078145.512704:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078145.512707:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5131 00000100:00080000:1.0:1769078145.512710:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078145.512713:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078145.512723:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078145.512741:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616859136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078145.512749:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616859008:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078145.512772:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078145.512778:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078145.512781:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078145.512787:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5131 00000100:00080000:1.0:1769078145.512805:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078145.512810:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078145.512818:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078145.512837:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616859264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078145.512851:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078145.512866:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078145.512869:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078145.512871:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078145.512880:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5131 00000100:00080000:1.0:1769078145.512884:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078145.512889:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078145.512896:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078145.512913:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078145/real 1769078145] req@ffff995b868c4700 x1855007616859008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078145.512913:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616859392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078145.512958:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078145.512973:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078145.512975:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078145.512981:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5131 00000100:00080000:1.0:1769078145.512984:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078145.512993:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078145.512995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616859008/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078145.512999:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078145.513012:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616859520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078145.513042:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078145.513044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078145.513047:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078145.513052:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078145.513061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769078145.513061:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5131 00000100:00080000:1.0:1769078145.513065:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078145.513072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616859008:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078145.513072:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078145.513078:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078145.513097:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616859648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078145.513108:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078145.513112:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078145.513115:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078145.513123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616859136:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078145.513124:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5131 00000100:00080000:1.0:1769078145.513136:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078145.513142:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078145.513148:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078145.513160:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616859776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078145.513175:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078145.513196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078145/real 1769078145] req@ffff995b868c5500 x1855007616859136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078145.513211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616859136/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078145.513220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078145.513225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078145.513228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616859136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078145.513236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616859264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078145.513256:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078145.513268:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078145/real 1769078145] req@ffff995b868c5c00 x1855007616859264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078145.513291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616859264/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078145.513303:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078145.513307:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078145.513311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616859264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078145.513318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616859392:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078145.513338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078145.513350:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078145/real 1769078145] req@ffff995b868c7800 x1855007616859392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078145.513362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616859392/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078145.513371:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078145.513375:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078145.513386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616859392:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078145.513394:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616859520:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078145.513414:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078145.513426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078145/real 1769078145] req@ffff995b868c7480 x1855007616859520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078145.513438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616859520/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078145.513449:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078145.513509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078145.513515:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616859520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078145.513530:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616859648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078145.513538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078145.513562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000800:00000100:0.0:1769078145.513563:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078145.513582:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078145/real 1769078145] req@ffff995b868c7100 x1855007616859648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078145.513598:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616859648/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078145.513608:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078145.513611:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078145.513614:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616859648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078145.513648:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616859776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078145.513671:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078145.513684:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078145/real 1769078145] req@ffff995b868c6a00 x1855007616859776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078145.513696:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616859776/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078145.513704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078145.513708:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078145.513711:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616859776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078146.216583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078146.216664:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078146.536528:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078146.536578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078146.536578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078146.536584:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078147.240617:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078147.240690:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078147.560626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078147.560626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078147.560672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078147.560707:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078148.264591:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078148.264604:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078148.584599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078148.584609:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078148.584637:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078148.584640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078149.288596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078149.288658:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078149.608566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078149.608611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078149.608613:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078149.608617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078150.312592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078150.312668:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078150.632571:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078150.632578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078150.632580:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078150.632587:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078150.632600:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5136 00000100:00100000:1.0:1769078150.632600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078150.632608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078150.632620:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078150.632632:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078150.632640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078150.632673:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616859904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078150.632695:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078150.632700:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078150.632702:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078150.632707:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5136 00000100:00080000:0.0:1769078150.632711:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078150.632716:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078150.632727:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078150.632732:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616859904:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078150.632745:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616860032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078150.632760:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078150.632763:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078150.632766:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078150.632770:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5136 00000100:00080000:0.0:1769078150.632774:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078150.632778:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078150.632782:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078150.632791:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616860160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078150.632803:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078150.632807:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078150.632809:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078150.632809:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078150.632817:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5136 00000100:00080000:0.0:1769078150.632819:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078150.632822:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078150.632828:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078150.632839:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616860288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078150.632849:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078150.632852:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078150.632855:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078150.632861:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5136 00000100:00100000:1.0:1769078150.632862:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078150/real 1769078150] req@ffff995b84641500 x1855007616859904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078150.632865:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078150.632873:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078150.632883:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078150.632895:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616859904/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078150.632896:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616860416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078150.632928:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078150.632931:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078150.632932:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078150.632933:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078150.632941:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5136 00000100:00080000:1.0:1769078150.632963:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078150.632971:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078150.632973:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616859904:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078150.632978:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078150.632984:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078150.632995:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616860544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078150.633001:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078150.633004:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078150.633005:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078150.633008:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5136 00000100:00080000:0.0:1769078150.633010:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078150.633013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860032:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769078150.633015:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078150.633022:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078150.633030:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616860672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078150.633053:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078150.633071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078150/real 1769078150] req@ffff995b84641180 x1855007616860032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078150.633088:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616860032/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078150.633099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078150.633103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078150.633107:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078150.633115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078150.633138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078150.633150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078150/real 1769078150] req@ffff995b84643800 x1855007616860160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078150.633165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616860160/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078150.633176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078150.633180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078150.633184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078150.633192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860288:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078150.633217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078150.633228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078150/real 1769078150] req@ffff995b84643480 x1855007616860288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078150.633243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616860288/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078150.633253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078150.633257:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078150.633261:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860288:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078150.633268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860416:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078150.633290:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078150.633303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078150/real 1769078150] req@ffff995b84642d80 x1855007616860416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078150.633318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616860416/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078150.633328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078150.633332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078150.633336:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078150.633342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616860544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078150.633365:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078150.633377:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078150/real 1769078150] req@ffff995b84643100 x1855007616860544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078150.633391:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616860544/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078150.633407:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078150.633412:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078150.633415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616860544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078150.633438:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616860672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078150.633485:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078150.633499:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078150/real 1769078150] req@ffff995b84643b80 x1855007616860672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078150.633514:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616860672/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078150.633524:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078150.633528:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078150.633532:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616860672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078151.336568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078151.336590:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078151.656573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078151.656604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078151.656616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078151.656620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078152.360572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078152.360659:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078152.680594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078152.680594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078152.680596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078152.680682:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078153.384615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078153.384677:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078153.704540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078153.704545:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078153.704545:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078153.704581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078154.408550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078154.408617:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078154.728575:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078154.728579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078154.728602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078154.728624:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078155.432581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078155.432654:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078155.752527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078155.752529:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078155.752554:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078155.752561:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078155.752566:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078155.752569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769078155.752576:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5141 00000100:00080000:0.0:1769078155.752588:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078155.752591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078155.752597:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078155.752639:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616860800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078155.752665:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078155.752667:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078155.752669:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078155.752673:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5141 00000100:00080000:0.0:1769078155.752676:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078155.752680:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078155.752688:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078155.752699:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616860800:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078155.752700:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616860928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078155.752718:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078155.752720:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078155.752721:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078155.752724:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5141 00000100:00080000:0.0:1769078155.752726:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078155.752728:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078155.752730:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078155.752738:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616861056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078155.752744:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078155.752746:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078155.752747:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078155.752750:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5141 00000100:00080000:0.0:1769078155.752765:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078155.752767:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078155.752771:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769078155.752776:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078155.752780:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616861184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078155.752787:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078155.752795:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078155.752796:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078155.752798:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5141 00000100:00080000:0.0:1769078155.752801:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078155.752803:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078155.752806:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078155.752814:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616861312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078155.752818:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078155/real 1769078155] req@ffff995b84642a00 x1855007616860800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078155.752825:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078155.752830:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078155.752832:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078155.752836:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5141 00000100:00080000:0.0:1769078155.752838:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078155.752841:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616860800/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078155.752843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078155.752847:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769078155.752866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769078155.752868:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616861440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078155.752875:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078155.752879:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078155.752883:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616860800:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078155.752884:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078155.752887:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078155.752893:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5141 00000100:00080000:0.0:1769078155.752896:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078155.752898:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078155.752902:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078155.752908:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616861568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078155.752913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078155.752935:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078155.752944:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078155/real 1769078155] req@ffff995b84640700 x1855007616860928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078155.752952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616860928/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078155.752960:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078155.752964:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078155.752967:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616860928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078155.752972:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861056:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078155.752990:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078155.753000:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078155/real 1769078155] req@ffff995b84641880 x1855007616861056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078155.753016:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616861056/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078155.753024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078155.753026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078155.753028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078155.753031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861184:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078155.753045:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078155.753051:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078155/real 1769078155] req@ffff995b84643b80 x1855007616861184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078155.753058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616861184/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078155.753063:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078155.753065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078155.753067:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861184:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078155.753070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861312:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078155.753078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078155.753083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078155/real 1769078155] req@ffff995b84643100 x1855007616861312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078155.753090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616861312/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078155.753099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078155.753101:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078155.753103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078155.753106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616861440:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078155.753118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078155.753127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078155/real 1769078155] req@ffff995b84642d80 x1855007616861440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078155.753136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616861440/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078155.753144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078155.753147:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078155.753150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616861440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078155.753169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616861568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078155.753187:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078155.753197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078155/real 1769078155] req@ffff995b84643480 x1855007616861568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078155.753209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616861568/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078155.753217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078155.753219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078155.753223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616861568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078156.456549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078156.456599:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078156.776561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078156.776561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078156.776561:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078156.776578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078157.480569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078157.480616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078157.800548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078157.800551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078157.800554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078157.800580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078158.504539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078158.504588:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078158.824560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078158.824560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078158.824560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078158.824560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078159.528563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078159.528627:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078159.848556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078159.848556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078159.848556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078159.848644:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078160.552518:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078160.552580:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078160.872569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078160.872570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078160.872672:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078160.872678:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078160.872679:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078160.872684:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078160.872693:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5146 00000100:00080000:3.0:1769078160.872703:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078160.872710:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078160.872727:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078160.872750:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616861696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078160.872784:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078160.872788:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078160.872790:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078160.872794:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5146 00000100:00080000:3.0:1769078160.872798:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078160.872802:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078160.872811:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078160.872824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616861824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078160.872833:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616861696:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078160.872838:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078160.872842:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078160.872843:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078160.872846:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5146 00000100:00080000:3.0:1769078160.872849:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078160.872851:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078160.872855:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078160.872865:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616861952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078160.872876:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078160.872877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078160.872878:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078160.872880:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5146 00000100:00080000:3.0:1769078160.872882:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078160.872884:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078160.872887:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078160.872893:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616862080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078160.872898:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078160.872900:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078160.872901:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078160.872903:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5146 00000100:00080000:3.0:1769078160.872906:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078160.872907:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078160.872911:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078160.872914:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078160.872920:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616862208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078160.872927:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078160.872929:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078160.872930:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078160.872933:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5146 00000100:00080000:3.0:1769078160.872935:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078160.872939:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078160.872942:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078160.872949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078160/real 1769078160] req@ffff995b84559500 x1855007616861696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078160.872950:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616862336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078160.872969:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078160.872974:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078160.872975:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078160.872979:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5146 00000100:00080000:3.0:1769078160.872981:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078160.872985:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078160.872987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616861696/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078160.872988:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078160.872998:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616862464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078160.873003:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078160.873010:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078160.873022:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616861696:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078160.873044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078160.873065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078160.873082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078160/real 1769078160] req@ffff995b8455aa00 x1855007616861824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078160.873094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616861824/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078160.873107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078160.873110:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078160.873113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078160.873119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078160.873134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078160.873144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078160/real 1769078160] req@ffff995b8455b100 x1855007616861952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078160.873168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616861952/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078160.873178:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078160.873181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078160.873184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616861952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078160.873189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862080:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078160.873206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078160.873214:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078160/real 1769078160] req@ffff995b8455a680 x1855007616862080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078160.873224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616862080/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078160.873232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078160.873235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078160.873238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862080:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078160.873243:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862208:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078160.873255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078160.873263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078160/real 1769078160] req@ffff995b84558a80 x1855007616862208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078160.873273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616862208/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078160.873281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078160.873284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078160.873287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862208:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078160.873291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616862336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078160.873304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078160.873312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078160/real 1769078160] req@ffff995b8455bb80 x1855007616862336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078160.873322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616862336/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078160.873330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078160.873333:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078160.873336:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616862336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078160.873341:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616862464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078160.873354:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078160.873362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078160/real 1769078160] req@ffff995b84558000 x1855007616862464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078160.873372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616862464/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078160.873379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078160.873382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078160.873385:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616862464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078161.576560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078161.576607:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078161.896563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078161.896563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078161.896703:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078161.896723:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078162.600557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078162.600596:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078162.920555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078162.920555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078162.920567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078162.920578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078163.624549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078163.624612:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078163.944581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078163.944597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078163.944597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078163.944610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078164.648566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078164.648654:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078164.968528:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078164.968530:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078164.968554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078164.968579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078165.672537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078165.672579:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078165.992560:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078165.992560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078165.992562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078165.992585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078165.992601:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078165.992607:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078165.992611:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078165.992620:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5151 00000100:00080000:0.0:1769078165.992630:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078165.992636:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078165.992672:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616862592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078165.992707:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078165.992710:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078165.992712:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078165.992715:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5151 00000100:00080000:0.0:1769078165.992718:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078165.992722:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078165.992731:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078165.992742:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616862592:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078165.992745:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616862720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078165.992755:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078165.992758:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078165.992759:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078165.992762:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5151 00000100:00080000:0.0:1769078165.992764:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078165.992766:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078165.992771:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078165.992780:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616862848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078165.992787:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078165.992789:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078165.992790:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078165.992792:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5151 00000100:00080000:0.0:1769078165.992801:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078165.992803:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078165.992807:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1769078165.992811:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078165.992820:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616862976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078165.992828:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078165.992830:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078165.992831:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078165.992834:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5151 00000100:00080000:0.0:1769078165.992836:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078165.992838:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078165.992842:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078165.992849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078165/real 1769078165] req@ffff995b84643800 x1855007616862592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078165.992853:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616863104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078165.992880:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078165.992882:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078165.992884:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078165.992888:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616862592/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078165.992890:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5151 00000100:00080000:0.0:1769078165.992895:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078165.992901:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078165.992908:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769078165.992910:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769078165.992917:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616863232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078165.992917:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078165.992923:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616862592:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078165.992931:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078165.992935:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078165.992936:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078165.992940:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5151 00000100:00080000:0.0:1769078165.992942:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078165.992944:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078165.992948:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078165.992950:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078165.992957:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616863360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078165.992977:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078165.992989:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078165/real 1769078165] req@ffff995b84641180 x1855007616862720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.992999:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616862720/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.993006:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.993009:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078165.993012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078165.993017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078165.993030:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078165.993039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078165/real 1769078165] req@ffff995b84641500 x1855007616862848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.993047:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616862848/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.993055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.993057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078165.993059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078165.993064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862976:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078165.993074:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078165.993079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078165/real 1769078165] req@ffff995b84640e00 x1855007616862976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.993091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616862976/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.993097:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.993099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078165.993100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616862976:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078165.993104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616863104:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078165.993113:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078165.993118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078165/real 1769078165] req@ffff995b84642680 x1855007616863104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.993125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616863104/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.993131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.993133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078165.993135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616863104:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078165.993138:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616863232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078165.993147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078165.993152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078165/real 1769078165] req@ffff995b84640000 x1855007616863232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.993159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616863232/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.993170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.993174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078165.993177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616863232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078165.993196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616863360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078165.993218:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078165.993230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078165/real 1769078165] req@ffff995b84640380 x1855007616863360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.993243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616863360/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.993249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.993251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078165.993254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616863360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078166.696579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078166.696699:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078167.016620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078167.016620:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078167.016620:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078167.016626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078167.720590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078167.720681:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078168.040640:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078168.040640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078168.040641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078168.040700:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078168.744587:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078168.744692:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078169.064587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078169.064608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078169.064611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078169.064612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078169.768555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078169.768670:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078170.088590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078170.088590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078170.088592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078170.088600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078170.792572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078170.792652:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078171.112542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078171.112546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078171.112547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078171.112593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078171.112617:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078171.112624:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078171.112628:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078171.112638:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5156 00000100:00080000:3.0:1769078171.112649:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078171.112656:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078171.112693:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616863488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078171.112715:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078171.112718:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078171.112719:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078171.112722:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5156 00000100:00080000:3.0:1769078171.112724:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078171.112727:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078171.112735:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078171.112745:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616863616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078171.112752:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616863488:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078171.112755:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078171.112760:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078171.112761:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078171.112764:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5156 00000100:00080000:3.0:1769078171.112766:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078171.112769:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078171.112772:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078171.112781:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616863744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078171.112792:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078171.112794:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078171.112796:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078171.112798:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5156 00000100:00080000:3.0:1769078171.112800:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078171.112802:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078171.112805:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078171.112813:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616863872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078171.112820:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078171.112826:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078171.112829:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078171.112831:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078171.112834:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5156 00000100:00080000:3.0:1769078171.112838:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078171.112840:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078171.112843:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078171.112857:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616864000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078171.112864:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078171.112867:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769078171.112869:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078171/real 1769078171] req@ffff995b84559c00 x1855007616863488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078171.112870:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078171.112879:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5156 00000100:00080000:3.0:1769078171.112885:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078171.112891:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078171.112898:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078171.112911:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616863488/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078171.112912:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616864128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078171.112944:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078171.112946:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078171.112948:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078171.112951:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078171.112955:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769078171.112958:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5156 00000100:00100000:1.0:1769078171.112963:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616863488:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078171.112963:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078171.112969:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078171.112973:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078171.112981:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616864256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078171.112990:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616863616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078171.113011:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078171.113023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078171/real 1769078171] req@ffff995b8455a300 x1855007616863616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078171.113032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616863616/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078171.113038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078171.113040:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078171.113043:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616863616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078171.113046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616863744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078171.113056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078171.113061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078171/real 1769078171] req@ffff995b8455ad80 x1855007616863744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078171.113069:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616863744/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078171.113081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078171.113083:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078171.113085:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616863744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078171.113088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616863872:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078171.113098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078171.113104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078171/real 1769078171] req@ffff995b8455b480 x1855007616863872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078171.113111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616863872/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078171.113116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078171.113118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078171.113120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616863872:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078171.113124:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864000:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078171.113132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078171.113138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078171/real 1769078171] req@ffff995b84558700 x1855007616864000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078171.113146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616864000/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078171.113151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078171.113153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078171.113156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078171.113160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616864128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078171.113169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078171.113174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078171/real 1769078171] req@ffff995b84559880 x1855007616864128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078171.113181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616864128/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078171.113187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078171.113189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078171.113191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616864128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078171.113201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616864256:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078171.113212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078171.113222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078171/real 1769078171] req@ffff995b84558000 x1855007616864256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078171.113235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616864256/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078171.113243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078171.113247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078171.113250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616864256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078171.816560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078171.816629:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078172.136599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078172.136599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078172.136602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078172.136605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078172.840561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078172.840650:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078173.160523:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078173.160526:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078173.160539:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078173.160556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078173.864530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078173.864593:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078174.184572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078174.184572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078174.184572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078174.184572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078174.888575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078174.888644:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078175.208567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078175.208574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078175.208583:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078175.208601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078175.912569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078175.912645:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078176.232587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078176.232587:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078176.232658:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078176.232661:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078176.232702:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078176.232709:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078176.232714:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078176.232724:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5161 00000100:00080000:2.0:1769078176.232735:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078176.232744:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078176.232784:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616864384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078176.232821:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078176.232825:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078176.232828:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078176.232833:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5161 00000100:00080000:2.0:1769078176.232839:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078176.232844:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078176.232854:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078176.232870:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616864512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078176.232874:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616864384:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078176.232893:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078176.232897:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078176.232899:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078176.232906:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5161 00000100:00080000:2.0:1769078176.232912:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078176.232918:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078176.232927:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078176.232945:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616864640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078176.232950:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078176.232964:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078176.232971:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078176.232976:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078176.232982:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5161 00000100:00080000:2.0:1769078176.232987:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078176.232991:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078176.232997:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078176.233001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078176/real 1769078176] req@ffff995b85ac9880 x1855007616864384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078176.233012:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616864768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078176.233021:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616864384/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078176.233029:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078176.233035:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078176.233036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078176.233038:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078176.233042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078176.233049:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616864384:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078176.233049:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5161 00000100:00080000:2.0:1769078176.233059:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078176.233066:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078176.233071:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078176.233078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078176.233086:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616864896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078176.233100:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078176.233104:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1769078176.233107:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078176.233108:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078176.233116:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5161 00000100:00080000:2.0:1769078176.233122:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078176.233125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078176/real 1769078176] req@ffff995b85ac8380 x1855007616864512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078176.233130:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078176.233141:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078176.233146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616864512/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078176.233153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769078176.233157:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616865024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078176.233158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078176.233166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078176.233189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864640:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078176.233193:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078176.233197:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078176.233200:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078176.233205:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5161 00000100:00080000:2.0:1769078176.233210:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078176.233213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078176.233218:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078176.233225:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078176.233232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078176/real 1769078176] req@ffff995b85acb100 x1855007616864640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078176.233241:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616865152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078176.233259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616864640/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078176.233277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078176.233282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078176.233301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078176.233316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864768:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078176.233345:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078176.233362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078176/real 1769078176] req@ffff995b85aca300 x1855007616864768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078176.233371:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616864768/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078176.233379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078176.233383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078176.233385:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864768:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078176.233398:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078176.233417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078176.233428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078176/real 1769078176] req@ffff995b85acb800 x1855007616864896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078176.233443:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616864896/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078176.233472:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078176.233478:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078176.233482:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616864896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078176.233488:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616865024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078176.233509:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078176.233534:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078176/real 1769078176] req@ffff995b85ac8a80 x1855007616865024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078176.233543:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616865024/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078176.233549:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078176.233551:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078176.233553:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616865024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078176.233557:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616865152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078176.233570:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078176.233577:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078176/real 1769078176] req@ffff995b85ac9c00 x1855007616865152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078176.233584:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616865152/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078176.233594:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078176.233597:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078176.233599:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616865152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078176.936572:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078176.936662:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078177.256552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078177.256552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078177.256563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078177.256593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078177.960551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078177.960631:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078178.280613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078178.280625:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078178.280671:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078178.280686:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078178.984585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078178.984665:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078179.304566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078179.304603:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078179.304612:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078179.304627:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078180.008528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078180.008602:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078180.328558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078180.328558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078180.328558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078180.328565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078181.032563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078181.032624:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078181.352527:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078181.352530:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078181.352540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078181.352545:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078181.352551:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078181.352555:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078181.352562:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5167 00000100:00080000:3.0:1769078181.352572:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078181.352578:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078181.352582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078181.352616:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616865280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078181.352639:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078181.352642:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078181.352644:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078181.352647:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5167 00000100:00080000:3.0:1769078181.352650:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078181.352653:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078181.352662:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078181.352668:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616865280:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078181.352673:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616865408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078181.352681:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078181.352683:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078181.352684:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078181.352686:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5167 00000100:00080000:3.0:1769078181.352688:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078181.352690:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078181.352693:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078181.352701:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616865536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078181.352707:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078181.352710:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078181.352712:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078181.352713:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5167 00000100:00080000:3.0:1769078181.352716:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078181.352718:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078181.352720:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078181.352728:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616865664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078181.352733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769078181.352737:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078181.352737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078181.352740:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078181.352744:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5167 00000100:00080000:3.0:1769078181.352746:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078181.352748:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078181.352751:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078181.352760:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616865792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078181.352768:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078181.352771:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078181.352773:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078181.352779:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078181/real 1769078181] req@ffff995b8455bb80 x1855007616865280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078181.352790:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5167 00000100:00080000:3.0:1769078181.352796:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078181.352799:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616865280/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078181.352803:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769078181.352813:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769078181.352813:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769078181.352819:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078181.352822:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616865280:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078181.352829:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616865920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078181.352838:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078181.352842:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769078181.352843:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616865408:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078181.352845:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078181.352852:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5167 00000100:00080000:3.0:1769078181.352857:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078181.352861:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769078181.352864:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769078181.352869:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078181.352875:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078181/real 1769078181] req@ffff995b84558a80 x1855007616865408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078181.352884:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616866048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078181.352897:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616865408/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078181.352905:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078181.352908:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078181.352911:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616865408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078181.352914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616865536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078181.352925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078181.352931:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078181/real 1769078181] req@ffff995b8455a680 x1855007616865536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078181.352937:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616865536/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078181.352943:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078181.352945:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078181.352946:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616865536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078181.352949:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616865664:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078181.352958:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078181.352963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078181/real 1769078181] req@ffff995b8455b100 x1855007616865664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078181.352969:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616865664/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078181.352979:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078181.352980:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078181.352982:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616865664:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078181.352985:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616865792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078181.352992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078181.352997:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078181/real 1769078181] req@ffff995b8455aa00 x1855007616865792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078181.353003:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616865792/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078181.353010:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078181.353013:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078181.353016:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616865792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078181.353021:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616865920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078181.353039:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078181.353049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078181/real 1769078181] req@ffff995b84559500 x1855007616865920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078181.353062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616865920/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078181.353070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078181.353072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078181.353074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616865920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078181.353086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616866048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078181.353099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078181.353105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078181/real 1769078181] req@ffff995b8455bb80 x1855007616866048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078181.353111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616866048/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078181.353116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078181.353118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078181.353120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616866048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078182.056575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078182.056655:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078182.376575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078182.376577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078182.376581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078182.376582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078183.080574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078183.080642:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078183.400574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078183.400590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078183.400591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078183.400670:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078184.104581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078184.104635:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078184.424552:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078184.424553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078184.424556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078184.424584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078185.128560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078185.128618:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078185.448573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078185.448573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078185.448574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078185.448580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078186.152554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078186.152627:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078186.472554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078186.472554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078186.472559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078186.472565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078186.472602:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078186.472607:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078186.472611:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078186.472618:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5172 00000100:00080000:2.0:1769078186.472627:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078186.472632:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078186.472664:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616866176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078186.472684:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078186.472687:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078186.472688:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078186.472690:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5172 00000100:00080000:2.0:1769078186.472693:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078186.472696:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078186.472703:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078186.472712:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616866304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078186.472716:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616866176:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078186.472722:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078186.472725:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078186.472726:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078186.472728:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5172 00000100:00080000:2.0:1769078186.472730:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078186.472733:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078186.472736:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078186.472746:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616866432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078186.472751:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078186.472754:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078186.472755:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078186.472756:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5172 00000100:00080000:2.0:1769078186.472758:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078186.472760:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078186.472763:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078186.472770:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616866560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078186.472774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078186.472781:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078186.472784:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078186.472785:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078186.472788:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5172 00000100:00080000:2.0:1769078186.472790:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078186.472793:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078186.472796:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078186.472803:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616866688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078186.472808:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078186.472812:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769078186.472813:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078186/real 1769078186] req@ffff995b85acaa00 x1855007616866176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078186.472814:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078186.472822:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5172 00000100:00080000:2.0:1769078186.472827:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078186.472830:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078186.472833:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078186.472835:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616866176/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078186.472842:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616866816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078186.472860:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078186.472861:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078186.472866:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078186.472868:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078186.472869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769078186.472874:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5172 00000100:00100000:1.0:1769078186.472876:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616866176:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078186.472878:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078186.472884:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078186.472887:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078186.472893:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616866944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078186.472902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616866304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078186.472918:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078186.472926:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078186/real 1769078186] req@ffff995b85acbb80 x1855007616866304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078186.472933:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616866304/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078186.472938:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078186.472940:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078186.472943:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616866304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078186.472946:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616866432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078186.472965:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078186.472971:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078186/real 1769078186] req@ffff995b85ac9c00 x1855007616866432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078186.472978:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616866432/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078186.472984:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078186.472986:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078186.472988:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616866432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078186.472992:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616866560:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078186.473002:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078186.473007:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078186/real 1769078186] req@ffff995b85ac8a80 x1855007616866560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078186.473014:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616866560/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078186.473025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078186.473026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078186.473028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616866560:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078186.473032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616866688:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078186.473041:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078186.473047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078186/real 1769078186] req@ffff995b85acb800 x1855007616866688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078186.473053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616866688/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078186.473058:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078186.473060:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078186.473061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616866688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078186.473065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616866816:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078186.473073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078186.473079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078186/real 1769078186] req@ffff995b85aca300 x1855007616866816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078186.473085:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616866816/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078186.473090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078186.473091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078186.473093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616866816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078186.473108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616866944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078186.473118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078186.473124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078186/real 1769078186] req@ffff995b85acb100 x1855007616866944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078186.473130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616866944/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078186.473135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078186.473137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078186.473138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616866944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078187.176576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078187.176996:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078187.496571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078187.496571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078187.496576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078187.496577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078188.200555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078188.200612:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078188.520553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078188.520553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078188.520560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078188.520561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078189.224549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078189.224623:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078189.544558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078189.544558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078189.544558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078189.544558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078190.248547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078190.248596:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078190.568535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078190.568535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078190.568545:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078190.568576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078191.272535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078191.272590:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078191.592631:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078191.592634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078191.592636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078191.592646:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078191.592651:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078191.592663:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5177 00000100:00100000:1.0:1769078191.592667:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078191.592670:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769078191.592673:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078191.592686:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078191.592723:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616867072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078191.592746:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078191.592749:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078191.592750:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078191.592753:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5177 00000100:00080000:2.0:1769078191.592755:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078191.592758:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078191.592766:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078191.592774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616867072:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078191.592777:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616867200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078191.592785:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078191.592787:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078191.592788:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078191.592790:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5177 00000100:00080000:2.0:1769078191.592793:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078191.592795:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078191.592797:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078191.592806:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616867328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078191.592812:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078191.592813:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078191.592814:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078191.592816:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5177 00000100:00080000:2.0:1769078191.592818:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078191.592820:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078191.592822:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078191.592828:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616867456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078191.592833:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078191.592835:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078191.592836:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078191.592837:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5177 00000800:00000100:1.0:1769078191.592845:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078191.592846:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078191.592852:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078191.592858:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078191.592870:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616867584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078191.592879:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078191.592880:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078191/real 1769078191] req@ffff995b85ac8380 x1855007616867072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078191.592888:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078191.592893:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078191.592902:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5177 00000100:00100000:1.0:1769078191.592903:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616867072/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078191.592911:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078191.592914:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078191.592919:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078191.592921:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616867072:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078191.592930:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078191.592937:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078191.592945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616867200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078191.592951:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616867712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078191.592962:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769078191.592964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078191.592967:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078191.592970:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078191.592974:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5177 00000100:00100000:1.0:1769078191.592977:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078191/real 1769078191] req@ffff995b85ac9880 x1855007616867200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078191.592980:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078191.592986:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078191.592995:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078191.592998:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616867200/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078191.593005:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078191.593008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078191.593012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616867200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078191.593015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616867328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078191.593019:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616867840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078191.593029:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078191.593040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078191/real 1769078191] req@ffff995b85acb100 x1855007616867328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078191.593053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616867328/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078191.593059:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078191.593061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078191.593063:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616867328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078191.593067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616867456:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078191.593077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078191.593082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078191/real 1769078191] req@ffff995b85aca300 x1855007616867456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078191.593089:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616867456/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078191.593094:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078191.593096:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078191.593098:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616867456:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078191.593100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616867584:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078191.593108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078191.593113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078191/real 1769078191] req@ffff995b85acb800 x1855007616867584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078191.593119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616867584/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078191.593129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078191.593131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078191.593132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616867584:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078191.593146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616867712:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078191.593164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078191.593174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078191/real 1769078191] req@ffff995b85ac8a80 x1855007616867712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078191.593185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616867712/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078191.593194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078191.593197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078191.593199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616867712:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078191.593203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616867840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078191.593216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078191.593223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078191/real 1769078191] req@ffff995b85ac9c00 x1855007616867840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078191.593230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616867840/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078191.593235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078191.593237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078191.593239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616867840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078192.296581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078192.296648:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078192.616544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078192.616561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078192.616592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078192.616601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078193.320543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078193.320633:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078193.640561:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078193.640565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078193.640603:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078193.640616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078194.344552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078194.344634:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078194.664522:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078194.664566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078194.664570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078194.664601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078195.368530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078195.368595:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078195.688565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078195.688565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078195.688575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078195.688596:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078196.392541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078196.392606:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078196.712510:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078196.712540:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078196.712540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078196.712548:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078196.712554:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078196.712562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078196.712563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078196.712564:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5182 00000100:00080000:0.0:1769078196.712577:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078196.712584:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078196.712622:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616867968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078196.712646:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078196.712650:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078196.712652:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078196.712656:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5182 00000100:00080000:0.0:1769078196.712660:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078196.712664:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078196.712673:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078196.712676:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616867968:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078196.712687:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616868096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078196.712699:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078196.712702:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078196.712705:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078196.712709:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5182 00000100:00080000:0.0:1769078196.712723:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078196.712727:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078196.712731:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078196.712743:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616868224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078196.712752:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078196.712755:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078196.712758:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078196.712762:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5182 00000100:00080000:0.0:1769078196.712766:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078196.712770:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868096:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769078196.712772:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078196.712782:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078196.712795:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616868352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078196.712805:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078196.712810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868224:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769078196.712811:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078196.712815:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078196.712821:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5182 00000100:00080000:0.0:1769078196.712826:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078196.712830:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078196.712835:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078196.712845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078196.712849:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616868480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078196.712861:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078196.712865:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078196.712867:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078196.712871:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5182 00000100:00080000:0.0:1769078196.712875:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078196.712881:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868480:192.168.204.132@tcp:8:kworker.0 00000100:00080000:0.0:1769078196.712882:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078196.712891:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078196.712904:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616868608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078196.712914:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078196.712918:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769078196.712920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616868608:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769078196.712921:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078196.712929:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5182 00000100:00080000:0.0:1769078196.712934:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078196.712938:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078196.712943:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078196.712954:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616868736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078196.712985:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616868736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078197.416515:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078197.416583:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078197.736562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078197.736566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078197.736617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078197.736641:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078198.440546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078198.440610:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078198.760549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078198.760558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078198.760566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078198.760593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078199.464534:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078199.464582:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078199.784566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078199.784566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078199.784566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078199.784721:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:1.0:1769078199.784739:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769078199.784778:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769078199.784795:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078199.784838:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078199.784846:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078199.784853:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078199.784860:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078199.784866:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078199.784874:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:0.0:1769078199.784933:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078196/real 1769078199] req@ffff995b84640e00 x1855007616868736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078199.784956:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616868736/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078199.784967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078199.784972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078199.784976:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616868736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078199.784983:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078196/real 1769078199] req@ffff995b84642680 x1855007616868608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078199.784994:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616868608/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078199.785001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078199.785004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078199.785007:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616868608:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078199.785013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078196/real 1769078199] req@ffff995b84640000 x1855007616868480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078199.785023:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616868480/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078199.785032:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078199.785036:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078199.785039:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078199.785044:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078196/real 1769078199] req@ffff995b84640380 x1855007616868352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078199.785056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616868352/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078199.785064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078199.785067:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078199.785070:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868352:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078199.785076:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078196/real 1769078199] req@ffff995b84640a80 x1855007616868096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078199.785088:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616868096/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078199.785095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078199.785099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078199.785102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078199.785108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078196/real 1769078199] req@ffff995b84641f80 x1855007616868224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078199.785145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616868224/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078199.785156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078199.785159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078199.785162:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078199.785169:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078196/real 1769078199] req@ffff995b84641c00 x1855007616867968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078199.785181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616867968/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078199.785188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078199.785191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078199.785194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616867968:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078200.488560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078200.488617:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078200.808570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078200.808570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078200.808680:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078200.808869:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078201.512562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078201.512620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078201.832575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078201.832575:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078201.832585:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1769078201.832589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078201.832590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078201.832593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078201.832602:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5187 00000100:00080000:3.0:1769078201.832613:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078201.832621:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078201.832622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078201.832659:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616868864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078201.832689:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078201.832693:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078201.832695:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078201.832699:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5187 00000100:00080000:3.0:1769078201.832701:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078201.832705:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078201.832714:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078201.832725:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616868992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078201.832733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078201.832736:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078201.832738:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078201.832742:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5187 00000100:00080000:3.0:1769078201.832745:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078201.832748:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078201.832752:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078201.832761:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616869120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078201.832769:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078201.832772:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078201.832774:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078201.832777:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5187 00000100:00080000:3.0:1769078201.832781:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078201.832785:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078201.832791:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078201.832802:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616868864:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078201.832803:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616869248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078201.832817:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078201.832821:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078201.832823:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078201.832826:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5187 00000100:00080000:3.0:1769078201.832830:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078201.832834:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078201.832838:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078201.832849:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616869376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078201.832858:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078201.832862:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078201.832864:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078201.832868:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5187 00000100:00080000:3.0:1769078201.832873:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078201.832875:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078201.832880:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078201.832885:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078201.832895:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616869504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078201.832904:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078201.832908:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078201.832909:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078201.832913:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5187 00000100:00080000:3.0:1769078201.832917:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078201.832921:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078201.832926:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078201.832927:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078201/real 1769078201] req@ffff995b84558000 x1855007616868864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078201.832940:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616869632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078201.832950:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616868864/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.832962:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.832967:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078201.832971:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616868864:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078201.832978:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078201.832998:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078201.833009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078201/real 1769078201] req@ffff995b84559880 x1855007616868992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078201.833020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616868992/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.833028:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.833031:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078201.833034:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616868992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078201.833040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616869120:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078201.833056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078201.833073:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078201/real 1769078201] req@ffff995b84558700 x1855007616869120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078201.833095:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616869120/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.833105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.833108:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078201.833111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616869120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078201.833139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616869248:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078201.833159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078201.833169:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078201/real 1769078201] req@ffff995b8455b480 x1855007616869248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078201.833186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616869248/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.833194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.833197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078201.833200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616869248:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078201.833205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616869376:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078201.833220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078201.833229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078201/real 1769078201] req@ffff995b8455ad80 x1855007616869376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078201.833239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616869376/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.833247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.833250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078201.833253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616869376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078201.833258:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616869504:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078201.833273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078201.833282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078201/real 1769078201] req@ffff995b8455a300 x1855007616869504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078201.833293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616869504/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.833306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.833310:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078201.833313:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616869504:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078201.833318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616869632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078201.833336:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078201.833348:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078201/real 1769078201] req@ffff995b84559c00 x1855007616869632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078201.833359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616869632/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.833367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.833370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078201.833374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616869632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078202.536551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078202.536616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078202.856558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078202.856559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078202.856667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078202.856818:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078203.560557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078203.560612:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078203.880573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078203.880584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078203.880606:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078203.880710:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078204.584549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078204.584609:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078204.904564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078204.904572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078204.904786:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078204.904969:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078205.608543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078205.608592:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078205.928536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078205.928548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078205.928682:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078205.928684:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078206.632546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078206.632604:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078206.952532:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078206.952538:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078206.952538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078206.952545:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078206.952549:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078206.952558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078206.952565:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5192 00000100:00080000:2.0:1769078206.952576:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078206.952583:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078206.952585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078206.952627:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616869760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078206.952650:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078206.952652:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078206.952653:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078206.952656:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5192 00000100:00080000:2.0:1769078206.952659:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078206.952662:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078206.952669:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078206.952687:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616869760:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078206.952688:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616869888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078206.952706:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078206.952708:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078206.952709:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078206.952712:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5192 00000100:00080000:2.0:1769078206.952714:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078206.952716:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078206.952719:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078206.952728:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616870016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078206.952734:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078206.952737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078206.952738:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078206.952740:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5192 00000100:00080000:2.0:1769078206.952742:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078206.952744:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078206.952747:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078206.952752:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616870144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078206.952758:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078206.952761:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078206.952762:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078206.952766:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5192 00000800:00000100:0.0:1769078206.952768:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078206.952770:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078206.952774:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078206.952777:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078206.952783:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616870272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078206.952791:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078206.952794:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078206.952796:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078206.952800:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5192 00000100:00080000:2.0:1769078206.952803:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078206.952807:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078206.952812:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078206.952827:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078206/real 1769078206] req@ffff995b85acbb80 x1855007616869760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078206.952828:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616870400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078206.952852:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078206.952856:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078206.952858:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078206.952864:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616869760/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078206.952864:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5192 00000100:00080000:2.0:1769078206.952870:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078206.952876:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078206.952883:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078206.952885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078206.952893:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078206.952894:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616870528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078206.952900:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616869760:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078206.952932:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616869888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078206.952956:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078206.952969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078206/real 1769078206] req@ffff995b85acaa00 x1855007616869888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078206.952982:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616869888/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078206.952990:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078206.953007:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078206.953011:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616869888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078206.953017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078206.953037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078206.953049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078206/real 1769078206] req@ffff995b85ac9c00 x1855007616870016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078206.953067:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616870016/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078206.953078:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078206.953081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078206.953084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078206.953090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870144:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078206.953108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078206.953118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078206/real 1769078206] req@ffff995b85ac8a80 x1855007616870144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078206.953130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616870144/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078206.953139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078206.953142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078206.953145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870144:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078206.953150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078206.953168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078206.953177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078206/real 1769078206] req@ffff995b85acb800 x1855007616870272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078206.953187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616870272/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078206.953201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078206.953204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078206.953207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078206.953212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616870400:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078206.953229:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078206.953239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078206/real 1769078206] req@ffff995b85aca300 x1855007616870400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078206.953250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616870400/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078206.953258:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078206.953262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078206.953265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616870400:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078206.953270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616870528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078206.953287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078206.953297:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078206/real 1769078206] req@ffff995b85acb100 x1855007616870528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078206.953308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616870528/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078206.953316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078206.953319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078206.953322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616870528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078207.656549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078207.656606:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078207.976547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078207.976550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078207.976557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078207.976566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078208.680542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078208.680612:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078209.000533:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078209.000533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078209.000571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078209.000731:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078209.704554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078209.704616:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078210.024525:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078210.024535:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078210.024619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078210.024619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078210.728563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078210.728629:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078211.048514:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078211.048539:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078211.048625:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078211.048639:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078211.752579:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078211.752657:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078212.072534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078212.072565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078212.072568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078212.072612:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078212.072620:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078212.072625:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078212.072629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769078212.072639:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5197 00000100:00080000:2.0:1769078212.072654:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078212.072665:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078212.072714:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616870656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078212.072746:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078212.072749:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078212.072751:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078212.072757:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5197 00000100:00080000:2.0:1769078212.072763:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078212.072768:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078212.072778:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078212.072793:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616870656:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078212.072799:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616870784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078212.072817:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078212.072821:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078212.072825:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078212.072831:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5197 00000100:00080000:2.0:1769078212.072837:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078212.072843:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078212.072849:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078212.072863:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616870912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078212.072873:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769078212.072884:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078212.072885:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078212.072890:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078212.072897:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5197 00000100:00080000:2.0:1769078212.072904:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078212.072925:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078212.072931:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078212.072951:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616871040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078212.072963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078212/real 1769078212] req@ffff995b85ac9880 x1855007616870656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078212.072964:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078212.072978:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078212.072986:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078212.072999:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5197 00000100:00100000:1.0:1769078212.073004:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616870656/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078212.073005:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078212.073020:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769078212.073026:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769078212.073030:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1769078212.073035:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078212.073044:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616870656:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078212.073056:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616871168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078212.073068:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078212.073072:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769078212.073075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870784:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078212.073077:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078212.073088:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5197 00000100:00080000:2.0:1769078212.073095:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078212.073101:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078212.073107:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078212.073116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078212.073121:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616871296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078212.073140:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078212.073145:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078212.073147:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078212.073149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078212/real 1769078212] req@ffff995b85ac8380 x1855007616870784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078212.073157:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5197 00000100:00080000:2.0:1769078212.073169:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078212.073181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616870784/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078212.073182:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078212.073201:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769078212.073202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078212.073213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078212.073216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078212.073226:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616871424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078212.073229:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870912:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078212.073274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078212.073301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078212/real 1769078212] req@ffff995b85acb100 x1855007616870912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078212.073316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616870912/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078212.073338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078212.073342:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078212.073346:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616870912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078212.073351:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871040:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078212.073366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078212.073374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078212/real 1769078212] req@ffff995b85aca300 x1855007616871040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078212.073381:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616871040/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078212.073388:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078212.073390:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078212.073393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871040:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078212.073399:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871168:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078212.073421:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078212.073433:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078212/real 1769078212] req@ffff995b85acb800 x1855007616871168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078212.073443:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616871168/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078212.073452:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078212.073494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078212.073498:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078212.073523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616871296:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078212.073543:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078212.073552:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078212/real 1769078212] req@ffff995b85ac8a80 x1855007616871296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078212.073563:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616871296/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078212.073571:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078212.073574:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078212.073577:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616871296:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078212.073583:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616871424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078212.073599:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078212.073609:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078212/real 1769078212] req@ffff995b85ac9c00 x1855007616871424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078212.073619:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616871424/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078212.073628:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078212.073631:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078212.073634:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616871424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078212.776560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078212.776620:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078213.096533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078213.096533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078213.096537:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078213.096560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078213.800588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078213.800650:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078214.120529:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078214.120545:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078214.120575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078214.120594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078214.824552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078214.824639:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078215.144587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078215.144591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078215.144645:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078215.144715:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078215.848556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078215.848635:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078216.168536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078216.168536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078216.168589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078216.168622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078216.872522:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078216.872602:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078217.192583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078217.192583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078217.192611:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078217.192617:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078217.192620:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078217.192627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769078217.192630:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5202 00000100:00100000:3.0:1769078217.192638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078217.192648:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078217.192656:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078217.192695:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616871552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078217.192716:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078217.192721:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078217.192723:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078217.192726:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5202 00000100:00080000:0.0:1769078217.192729:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078217.192732:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078217.192740:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078217.192748:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616871680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078217.192756:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616871552:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078217.192757:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078217.192762:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078217.192765:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078217.192769:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5202 00000100:00080000:0.0:1769078217.192771:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078217.192773:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078217.192775:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078217.192782:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616871808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078217.192787:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078217.192789:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078217.192790:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078217.192793:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5202 00000100:00080000:0.0:1769078217.192794:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078217.192796:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078217.192809:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078217.192819:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616871936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078217.192841:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078217.192845:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078217.192847:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078217.192851:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078217.192854:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5202 00000100:00080000:0.0:1769078217.192859:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078217.192864:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078217.192869:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078217.192881:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616872064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078217.192887:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078217.192889:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078217.192891:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078217.192894:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5202 00000100:00080000:0.0:1769078217.192897:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078217.192902:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078217.192903:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078217/real 1769078217] req@ffff995b84641c00 x1855007616871552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078217.192905:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078217.192916:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616872192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078217.192924:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078217.192926:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078217.192928:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078217.192935:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5202 00000100:00080000:0.0:1769078217.192940:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078217.192942:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078217.192944:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078217.192950:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616872320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078217.192955:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616871552/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078217.192967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078217.192972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078217.192977:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616871552:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078217.193005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871680:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078217.193028:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078217.193041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078217/real 1769078217] req@ffff995b84641f80 x1855007616871680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078217.193054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616871680/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078217.193062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078217.193065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078217.193069:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078217.193075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871808:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078217.193093:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078217.193103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078217/real 1769078217] req@ffff995b84640a80 x1855007616871808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078217.193115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616871808/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078217.193131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078217.193134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078217.193137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078217.193143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871936:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078217.193162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078217.193172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078217/real 1769078217] req@ffff995b84640380 x1855007616871936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078217.193183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616871936/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078217.193192:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078217.193195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078217.193199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616871936:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078217.193205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872064:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078217.193222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078217.193233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078217/real 1769078217] req@ffff995b84640000 x1855007616872064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078217.193244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616872064/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078217.193253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078217.193256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078217.193261:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872064:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078217.193266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616872192:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078217.193284:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078217.193295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078217/real 1769078217] req@ffff995b84642680 x1855007616872192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078217.193306:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616872192/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078217.193315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078217.193318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078217.193322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616872192:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078217.193327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616872320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078217.193345:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078217.193356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078217/real 1769078217] req@ffff995b84640e00 x1855007616872320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078217.193368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616872320/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078217.193376:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078217.193378:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078217.193382:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616872320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078217.896581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078217.896657:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078218.216629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078218.216630:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078218.216649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078218.216655:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078218.920523:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078218.920613:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078219.240564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078219.240565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078219.240565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078219.240571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078219.944569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078219.944731:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078220.264588:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078220.264588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078220.264617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078220.264633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078220.968580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078220.968664:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078221.288604:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078221.288604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078221.288604:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078221.288610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078221.992577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078221.992647:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078222.312615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078222.312615:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078222.312626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078222.312655:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078222.312661:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078222.312669:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078222.312676:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078222.312687:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5208 00000100:00080000:0.0:1769078222.312698:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078222.312706:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078222.312759:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616872448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078222.312784:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078222.312788:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078222.312790:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078222.312795:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5208 00000100:00080000:0.0:1769078222.312798:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078222.312804:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078222.312813:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078222.312829:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616872576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078222.312830:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616872448:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078222.312845:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078222.312850:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078222.312851:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078222.312856:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5208 00000100:00080000:0.0:1769078222.312859:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078222.312863:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078222.312880:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078222.312892:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616872704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078222.312902:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078222.312906:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1769078222.312909:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078222.312910:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078222.312916:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5208 00000100:00080000:0.0:1769078222.312920:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078222.312924:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078222.312929:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078222.312940:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616872832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078222.312949:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078222.312953:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078222.312955:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078222.312960:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078222/real 1769078222] req@ffff995b84641500 x1855007616872448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078222.312961:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5208 00000100:00080000:0.0:1769078222.312967:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078222.312975:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078222.312980:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078222.312990:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616872960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078222.312997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616872448/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078222.313003:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078222.313013:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078222.313015:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078222.313016:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078222.313022:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5208 00000100:00080000:1.0:1769078222.313023:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078222.313028:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078222.313030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616872448:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078222.313035:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078222.313041:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078222.313052:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616873088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078222.313060:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872576:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769078222.313063:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078222.313068:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078222.313070:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078222.313075:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5208 00000100:00080000:0.0:1769078222.313083:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078222.313088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078222.313090:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078222.313098:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078222.313106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078222/real 1769078222] req@ffff995b84641180 x1855007616872576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078222.313110:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616873216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078222.313138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616872576/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078222.313147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078222.313151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078222.313154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078222.313160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872704:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078222.313177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078222.313187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078222/real 1769078222] req@ffff995b84643800 x1855007616872704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078222.313197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616872704/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078222.313206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078222.313209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078222.313212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078222.313217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872832:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078222.313232:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078222.313240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078222/real 1769078222] req@ffff995b84643480 x1855007616872832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078222.313251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616872832/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078222.313259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078222.313262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078222.313265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872832:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078222.313270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872960:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078222.313284:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078222.313292:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078222/real 1769078222] req@ffff995b84642d80 x1855007616872960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078222.313303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616872960/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078222.313311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078222.313314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078222.313317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616872960:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078222.313338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616873088:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078222.313354:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078222.313363:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078222/real 1769078222] req@ffff995b84643100 x1855007616873088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078222.313373:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616873088/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078222.313380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078222.313384:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078222.313386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616873088:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078222.313391:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616873216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078222.313405:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078222.313413:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078222/real 1769078222] req@ffff995b84643b80 x1855007616873216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078222.313422:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616873216/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078222.313434:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078222.313437:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078222.313440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616873216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078223.016598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078223.016670:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078223.336584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078223.336590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078223.336590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078223.336767:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078224.040549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078224.042808:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078224.360494:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078224.360588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078224.360588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078224.360600:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078225.064564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078225.064637:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078225.384549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078225.384549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078225.384554:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078225.384637:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078226.088573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078226.088690:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078226.408602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078226.408605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078226.408761:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078226.408778:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078227.112561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078227.112632:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078227.432562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078227.432562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769078227.432575:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078227.432582:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078227.432586:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078227.432593:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5213 00000100:00100000:0.0:1769078227.432596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078227.432596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078227.432605:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078227.432612:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078227.432671:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616873344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078227.432700:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078227.432704:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078227.432706:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078227.432709:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5213 00000100:00080000:1.0:1769078227.432712:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078227.432715:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078227.432723:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078227.432733:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616873472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078227.432739:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616873344:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078227.432744:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078227.432748:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078227.432749:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078227.432753:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5213 00000100:00080000:1.0:1769078227.432756:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078227.432758:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078227.432761:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078227.432768:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616873600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078227.432774:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078227.432777:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078227.432778:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078227.432780:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5213 00000100:00080000:1.0:1769078227.432788:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078227.432790:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078227.432793:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769078227.432797:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078227.432802:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616873728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078227.432809:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078227.432812:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078227.432813:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078227.432815:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5213 00000100:00080000:1.0:1769078227.432818:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078227.432820:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078227.432823:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078227.432831:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078227/real 1769078227] req@ffff995b868c4a80 x1855007616873344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078227.432832:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616873856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078227.432857:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078227.432860:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078227.432861:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078227.432865:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5213 00000100:00100000:0.0:1769078227.432866:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616873344/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078227.432867:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078227.432873:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078227.432879:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078227.432882:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078227.432888:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078227.432888:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616873984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078227.432895:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616873344:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078227.432901:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078227.432904:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078227.432905:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078227.432908:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5213 00000100:00080000:1.0:1769078227.432910:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078227.432912:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078227.432914:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078227.432920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616873472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078227.432922:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616874112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078227.432941:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078227.432951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078227/real 1769078227] req@ffff995b868c6d80 x1855007616873472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078227.432963:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616873472/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078227.432968:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078227.432970:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078227.432972:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616873472:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078227.432975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616873600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078227.432984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078227.432989:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078227/real 1769078227] req@ffff995b868c6300 x1855007616873600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078227.432995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616873600/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078227.433001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078227.433003:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078227.433005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616873600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078227.433008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616873728:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078227.433017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078227.433022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078227/real 1769078227] req@ffff995b868c6a00 x1855007616873728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078227.433029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616873728/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078227.433038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078227.433039:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078227.433041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616873728:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078227.433044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616873856:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078227.433052:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078227.433057:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078227/real 1769078227] req@ffff995b868c7100 x1855007616873856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078227.433063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616873856/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078227.433068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078227.433070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078227.433071:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616873856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078227.433074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616873984:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078227.433083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078227.433088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078227/real 1769078227] req@ffff995b868c7480 x1855007616873984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078227.433094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616873984/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078227.433099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078227.433100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078227.433102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616873984:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078227.433112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616874112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078227.433121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078227.433127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078227/real 1769078227] req@ffff995b868c7800 x1855007616874112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078227.433133:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616874112/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078227.433137:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078227.433139:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078227.433140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616874112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078228.136545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078228.136612:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078228.456589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078228.456589:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078228.456589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078228.456598:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078229.160560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078229.160615:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078229.480559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078229.480559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078229.480565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078229.480583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078230.184623:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078230.184709:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078230.504621:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078230.504621:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078230.504627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078230.504733:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078231.208602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078231.208681:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078231.528562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078231.528596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078231.528617:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078231.528631:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078232.232583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078232.232637:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078232.552587:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078232.552587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078232.552587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078232.552588:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078232.552599:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078232.552604:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078232.552614:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5218 00000100:00100000:1.0:1769078232.552626:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769078232.552627:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078232.552637:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078232.552679:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616874240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078232.552724:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078232.552729:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078232.552731:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078232.552735:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5218 00000100:00080000:2.0:1769078232.552739:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078232.552742:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078232.552748:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616874240:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769078232.552750:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078232.552764:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616874368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078232.552776:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078232.552778:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078232.552780:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078232.552783:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5218 00000100:00080000:2.0:1769078232.552785:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078232.552787:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078232.552790:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078232.552799:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616874496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078232.552807:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078232.552808:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078232.552813:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078232.552814:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078232.552817:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5218 00000100:00080000:2.0:1769078232.552819:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078232.552822:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078232.552825:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078232.552832:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616874624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078232.552837:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078232.552839:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078232.552841:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078232.552843:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5218 00000100:00080000:2.0:1769078232.552846:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078232.552849:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078232.552852:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078232/real 1769078232] req@ffff995b85acaa00 x1855007616874240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769078232.552854:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078232.552866:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616874752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078232.552874:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616874240/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078232.552874:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078232.552881:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078232.552884:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078232.552891:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5218 00000100:00080000:2.0:1769078232.552895:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078232.552901:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078232.552903:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769078232.552904:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078232.552910:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078232.552917:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616874240:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078232.552917:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616874880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078232.552926:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078232.552930:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078232.552931:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078232.552935:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5218 00000100:00080000:2.0:1769078232.552937:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078232.552939:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078232.552942:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078232.552948:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616875008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078232.552952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616874368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078232.552983:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078232.552997:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078232/real 1769078232] req@ffff995b85acbb80 x1855007616874368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078232.553008:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616874368/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078232.553014:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078232.553016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078232.553018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616874368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078232.553022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616874496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078232.553035:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078232.553041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078232/real 1769078232] req@ffff995b85ac9c00 x1855007616874496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078232.553048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616874496/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078232.553055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078232.553058:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078232.553059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616874496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078232.553064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616874624:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078232.553073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078232.553078:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078232/real 1769078232] req@ffff995b85ac8a80 x1855007616874624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078232.553085:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616874624/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078232.553090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078232.553092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078232.553094:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616874624:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078232.553096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616874752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078232.553108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078232.553117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078232/real 1769078232] req@ffff995b85acb800 x1855007616874752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078232.553127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616874752/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078232.553135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078232.553138:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078232.553141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616874752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078232.553147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616874880:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078232.553163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078232.553173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078232/real 1769078232] req@ffff995b85aca300 x1855007616874880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078232.553184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616874880/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078232.553189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078232.553191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078232.553193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616874880:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078232.553206:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616875008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078232.553218:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078232.553224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078232/real 1769078232] req@ffff995b85acb100 x1855007616875008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078232.553230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616875008/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078232.553237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078232.553240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078232.553243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616875008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078233.256549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078233.256662:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078233.576540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078233.576540:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078233.576559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078233.576586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078234.280565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078234.280640:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078234.600590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078234.600591:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078234.600594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078234.600608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078235.304567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078235.304632:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078235.624535:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078235.624540:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078235.624553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078235.624570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078236.328532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078236.328611:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078236.648564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078236.648564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078236.648564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078236.648578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078237.352568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078237.352644:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078237.672560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078237.672560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078237.672562:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078237.672591:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078237.672591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769078237.672598:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078237.672603:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078237.672610:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5223 00000100:00080000:0.0:1769078237.672619:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078237.672625:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078237.672658:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616875136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078237.672678:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078237.672681:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078237.672682:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078237.672685:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5223 00000100:00080000:0.0:1769078237.672688:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078237.672697:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078237.672706:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616875136:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769078237.672707:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078237.672718:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616875264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078237.672729:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078237.672732:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078237.672735:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078237.672738:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5223 00000100:00080000:0.0:1769078237.672754:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078237.672759:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:1.0:1769078237.672763:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769078237.672767:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078237.672781:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616875392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078237.672789:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078237.672792:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078237.672794:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078237.672798:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5223 00000100:00080000:0.0:1769078237.672803:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078237.672807:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078237/real 1769078237] req@ffff995b84641880 x1855007616875136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078237.672810:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078237.672820:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078237.672828:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616875136/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078237.672835:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616875520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078237.672836:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078237.672844:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078237.672849:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078237.672850:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616875136:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078237.672856:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078237.672860:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078237.672864:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5223 00000100:00080000:0.0:1769078237.672867:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078237.672870:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078237.672873:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078237.672879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616875264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078237.672883:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616875648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078237.672891:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078237.672893:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078237.672894:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078237.672898:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5223 00000800:00000100:1.0:1769078237.672902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078237.672904:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078237.672908:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078237.672912:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078237.672916:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078237/real 1769078237] req@ffff995b84640700 x1855007616875264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078237.672923:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616875776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078237.672928:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616875264/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078237.672932:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078237.672939:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078237.672941:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078237.672945:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078237.672946:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5223 00000100:00080000:1.0:1769078237.672950:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078237.672951:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078237.672955:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616875264:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769078237.672956:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078237.672963:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078237.672966:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616875392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078237.672972:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616875904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078237.672981:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078237.672989:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078237/real 1769078237] req@ffff995b84642a00 x1855007616875392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078237.673003:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616875392/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078237.673013:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078237.673015:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078237.673017:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616875392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078237.673020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616875520:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078237.673030:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078237.673036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078237/real 1769078237] req@ffff995b84643b80 x1855007616875520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078237.673043:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616875520/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078237.673048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078237.673050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078237.673051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616875520:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078237.673063:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616875648:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078237.673079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078237.673089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078237/real 1769078237] req@ffff995b84643100 x1855007616875648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078237.673101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616875648/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078237.673109:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078237.673111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078237.673112:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616875648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078237.673116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616875776:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078237.673129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078237.673136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078237/real 1769078237] req@ffff995b84642d80 x1855007616875776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078237.673142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616875776/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078237.673151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078237.673152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078237.673154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616875776:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078237.673157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616875904:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078237.673165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078237.673170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078237/real 1769078237] req@ffff995b84643480 x1855007616875904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078237.673175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616875904/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078237.673180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078237.673181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078237.673183:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616875904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078238.376539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078238.376594:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078238.696557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078238.696557:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078238.696557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078238.696560:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078239.400549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078239.400620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078239.720558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078239.720558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078239.720558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078239.720558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078240.424549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078240.424646:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078240.744567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078240.744567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078240.744589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078240.744633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078241.448562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078241.448618:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078241.768570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078241.768570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078241.768570:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078241.768595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078242.472552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078242.472621:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078242.792556:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078242.792559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078242.792564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078242.792569:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078242.792574:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078242.792582:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5228 00000100:00100000:0.0:1769078242.792583:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078242.792591:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078242.792597:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078242.792630:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616876032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078242.792651:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078242.792653:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078242.792654:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078242.792657:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5228 00000100:00080000:3.0:1769078242.792659:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078242.792663:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078242.792672:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078242.792685:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616876032:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078242.792686:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616876160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078242.792698:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078242.792702:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078242.792704:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078242.792708:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5228 00000100:00080000:3.0:1769078242.792711:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078242.792714:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078242.792719:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078242.792729:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616876288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078242.792736:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078242.792738:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078242.792739:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078242.792742:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5228 00000100:00080000:3.0:1769078242.792744:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078242.792746:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078242.792748:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078242.792754:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616876416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078242.792760:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078242.792762:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078242.792764:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769078242.792766:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078242.792769:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5228 00000100:00080000:3.0:1769078242.792772:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078242.792775:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078242.792778:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078242.792789:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616876544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078242.792798:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078242.792801:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1769078242.792804:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078242/real 1769078242] req@ffff995b84559c00 x1855007616876032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078242.792805:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078242.792814:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5228 00000100:00080000:3.0:1769078242.792820:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078242.792823:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616876032/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078242.792828:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:0.0:1769078242.792834:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769078242.792836:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078242.792840:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078242.792844:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616876032:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078242.792849:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616876672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078242.792856:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078242.792858:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078242.792860:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078242.792863:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5228 00000100:00100000:0.0:1769078242.792866:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616876160:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078242.792867:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078242.792873:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078242.792876:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078242.792886:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616876800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078242.792890:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078242.792905:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078242/real 1769078242] req@ffff995b8455a300 x1855007616876160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078242.792914:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616876160/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078242.792919:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078242.792922:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078242.792923:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616876160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078242.792927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616876288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078242.792936:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078242.792942:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078242/real 1769078242] req@ffff995b8455ad80 x1855007616876288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078242.792953:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616876288/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078242.792970:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078242.792974:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078242.792978:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616876288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078242.792985:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616876416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078242.793003:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000800:00000100:0.0:1769078242.793017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078242.793031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078242/real 1769078242] req@ffff995b8455b480 x1855007616876416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078242.793042:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616876416/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078242.793049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078242.793052:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078242.793055:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616876416:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078242.793060:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616876544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078242.793076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078242.793086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078242/real 1769078242] req@ffff995b84558700 x1855007616876544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078242.793095:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616876544/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078242.793103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078242.793105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078242.793108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616876544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078242.793114:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616876672:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078242.793130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078242.793141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078242/real 1769078242] req@ffff995b84559880 x1855007616876672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078242.793151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616876672/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078242.793157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078242.793159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078242.793161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616876672:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078242.793173:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616876800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078242.793186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078242.793192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078242/real 1769078242] req@ffff995b84558000 x1855007616876800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078242.793199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616876800/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078242.793204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078242.793206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078242.793207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616876800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078243.496567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078243.496646:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078243.816565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078243.816626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078243.816628:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078243.816640:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078244.520594:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078244.520666:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078244.840583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078244.840584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078244.840605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078244.840611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078245.544599:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078245.544664:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078245.864557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078245.864558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078245.864590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078245.864618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078246.568565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078246.568649:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078246.888546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078246.888577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078246.888580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078246.888595:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078247.592560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078247.592621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078247.912549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769078247.912587:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078247.912587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078247.912592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078247.912596:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078247.912603:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078247.912625:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5233 00000100:00080000:0.0:1769078247.912637:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078247.912645:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078247.912658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078247.912683:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616876928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078247.912711:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078247.912715:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078247.912717:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078247.912722:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5233 00000100:00080000:0.0:1769078247.912726:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078247.912732:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078247.912741:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078247.912745:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616876928:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078247.912757:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616877056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078247.912769:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078247.912773:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078247.912775:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078247.912780:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5233 00000100:00080000:0.0:1769078247.912784:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078247.912788:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769078247.912810:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769078247.912816:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078247.912830:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616877184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078247.912842:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078247.912846:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078247.912848:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078247.912852:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5233 00000100:00080000:0.0:1769078247.912856:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078247.912861:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078247.912868:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078247/real 1769078247] req@ffff995b84643800 x1855007616876928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078247.912875:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078247.912883:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616876928/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078247.912889:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616877312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078247.912893:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078247.912903:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078247.912905:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078247.912909:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616876928:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078247.912911:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078247.912914:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078247.912920:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5233 00000100:00080000:0.0:1769078247.912924:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078247.912927:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078247.912932:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078247.912940:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078247.912947:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616877440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078247.912960:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078247.912964:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078247.912969:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769078247.912969:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078247.912976:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5233 00000100:00080000:0.0:1769078247.912981:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078247.912986:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078247.912992:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078247.912993:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078247/real 1769078247] req@ffff995b84641180 x1855007616877056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078247.913005:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616877568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078247.913011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616877056/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078247.913020:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078247.913027:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078247.913029:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078247.913030:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078247.913037:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078247.913039:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5233 00000100:00100000:2.0:1769078247.913044:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877056:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769078247.913046:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769078247.913050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877184:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769078247.913054:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078247.913062:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078247.913075:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616877696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078247.913082:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078247.913103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078247/real 1769078247] req@ffff995b84641500 x1855007616877184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078247.913118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616877184/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078247.913133:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078247.913136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078247.913139:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078247.913145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877312:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078247.913164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078247.913179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078247/real 1769078247] req@ffff995b84640e00 x1855007616877312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078247.913192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616877312/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078247.913202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078247.913206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078247.913211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877312:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078247.913226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078247.913249:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078247.913256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078247/real 1769078247] req@ffff995b84642680 x1855007616877440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078247.913263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616877440/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078247.913268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078247.913270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078247.913273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078247.913276:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616877568:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078247.913285:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078247.913291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078247/real 1769078247] req@ffff995b84640000 x1855007616877568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078247.913297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616877568/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078247.913302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078247.913327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078247.913330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616877568:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078247.913334:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616877696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078247.913351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078247.913360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078247/real 1769078247] req@ffff995b84640380 x1855007616877696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078247.913372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616877696/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078247.913380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078247.913383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078247.913386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616877696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078248.616548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078248.616614:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078248.936550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078248.936550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078248.936555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078248.936579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078249.640550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078249.640602:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078249.960578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078249.960582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078249.960752:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078249.960766:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078250.664567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078250.664647:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078250.984534:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078250.984534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078250.984542:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078250.984568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078251.688549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078251.688611:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078252.008556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078252.008556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078252.008560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078252.008615:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078252.712557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078252.712615:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078253.032535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078253.032536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078253.032541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078253.032564:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078253.032569:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078253.032575:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078253.032576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078253.032587:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5238 00000100:00080000:3.0:1769078253.032599:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078253.032607:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078253.032644:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616877824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078253.032664:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078253.032666:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078253.032667:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078253.032670:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5238 00000100:00080000:3.0:1769078253.032673:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078253.032677:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078253.032691:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078253.032693:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616877824:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078253.032701:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616877952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078253.032709:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078253.032712:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078253.032713:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078253.032716:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5238 00000100:00080000:3.0:1769078253.032718:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078253.032721:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078253.032724:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078253.032731:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616878080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078253.032737:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078253.032739:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078253.032741:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078253.032743:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5238 00000100:00080000:3.0:1769078253.032744:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078253.032747:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769078253.032751:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769078253.032753:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078253.032762:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616878208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078253.032768:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078253.032770:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078253.032771:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078253.032773:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5238 00000100:00080000:3.0:1769078253.032775:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078253.032778:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078253.032781:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078253.032785:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078253/real 1769078253] req@ffff995b8455bb80 x1855007616877824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078253.032789:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616878336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078253.032795:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078253.032802:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078253.032806:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078253.032813:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5238 00000100:00080000:3.0:1769078253.032817:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769078253.032818:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616877824/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078253.032826:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769078253.032831:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769078253.032834:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769078253.032840:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078253.032845:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616877824:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078253.032849:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616878464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078253.032859:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078253.032862:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078253.032864:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078253.032868:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5238 00000100:00080000:3.0:1769078253.032872:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769078253.032873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877952:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078253.032879:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078253.032886:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078253.032899:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616878592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078253.032910:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078253.032922:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078253/real 1769078253] req@ffff995b84559500 x1855007616877952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078253.032933:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616877952/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078253.032943:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078253.032945:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078253.032947:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616877952:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078253.032951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878080:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078253.032962:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078253.032967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078253/real 1769078253] req@ffff995b8455aa00 x1855007616878080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078253.032974:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616878080/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078253.032980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078253.032981:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078253.032983:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078253.032987:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878208:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078253.033004:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078253.033013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078253/real 1769078253] req@ffff995b8455b100 x1855007616878208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078253.033024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616878208/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078253.033031:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078253.033034:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078253.033038:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878208:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078253.033043:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878336:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078253.033061:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078253.033071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078253/real 1769078253] req@ffff995b8455a680 x1855007616878336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078253.033081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616878336/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078253.033089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078253.033092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078253.033095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078253.033100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616878464:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078253.033118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078253.033127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078253/real 1769078253] req@ffff995b84558a80 x1855007616878464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078253.033134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616878464/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078253.033139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078253.033141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078253.033143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616878464:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078253.033155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616878592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078253.033165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078253.033171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078253/real 1769078253] req@ffff995b8455bb80 x1855007616878592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078253.033177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616878592/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078253.033182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078253.033184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078253.033185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616878592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078253.736576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078253.736677:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078254.056579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078254.056579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078254.056580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078254.056593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078254.760600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078254.760678:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078255.080558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078255.080567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078255.080623:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078255.080637:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078255.784561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078255.784616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078256.104612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078256.104613:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078256.104618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078256.104627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078256.808595:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078256.808656:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078257.128568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078257.128580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078257.128660:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078257.128687:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078257.832556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078257.832569:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078258.152579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078258.152579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078258.152611:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078258.152615:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078258.152623:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078258.152627:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078258.152637:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5243 00000100:00100000:3.0:1769078258.152638:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078258.152648:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078258.152655:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078258.152691:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616878720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078258.152713:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078258.152715:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078258.152717:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078258.152719:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5243 00000100:00080000:0.0:1769078258.152721:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078258.152723:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078258.152731:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078258.152740:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616878848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078258.152746:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078258.152748:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078258.152749:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078258.152752:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5243 00000100:00080000:0.0:1769078258.152754:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078258.152759:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078258.152763:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616878720:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769078258.152767:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078258.152783:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616878976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078258.152795:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078258.152800:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078258.152802:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078258.152806:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5243 00000100:00080000:0.0:1769078258.152811:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078258.152815:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078258.152820:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078258.152829:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616879104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078258.152835:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078258.152837:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078258.152838:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078258.152843:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5243 00000800:00000100:2.0:1769078258.152849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078258.152852:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078258.152857:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078258.152863:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078258.152875:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616879232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078258.152882:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078258.152884:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078258.152887:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078258.152892:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5243 00000100:00080000:0.0:1769078258.152894:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078258.152896:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078258.152899:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078258.152906:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616879360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078258.152911:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078258/real 1769078258] req@ffff995b84640a80 x1855007616878720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078258.152926:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078258.152932:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078258.152935:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078258.152937:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616878720/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078258.152947:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5243 00000100:00080000:2.0:1769078258.152952:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078258.152955:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078258.152961:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078258.152965:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616878720:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078258.152968:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078258.152983:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078258.152988:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078258.153004:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616879488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078258.153011:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078258.153022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078258/real 1769078258] req@ffff995b84641f80 x1855007616878848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078258.153041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616878848/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078258.153047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078258.153049:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078258.153052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878848:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078258.153073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078258.153084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078258.153090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078258/real 1769078258] req@ffff995b84641c00 x1855007616878976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078258.153096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616878976/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078258.153103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078258.153105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078258.153106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616878976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078258.153109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616879104:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078258.153119:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078258.153124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078258/real 1769078258] req@ffff995b84640380 x1855007616879104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078258.153142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616879104/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078258.153148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078258.153150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078258.153152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616879104:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078258.153156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616879232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078258.153168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078258.153175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078258/real 1769078258] req@ffff995b84640000 x1855007616879232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078258.153181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616879232/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078258.153187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078258.153188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078258.153190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616879232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078258.153193:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616879360:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078258.153203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078258.153208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078258/real 1769078258] req@ffff995b84642680 x1855007616879360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078258.153218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616879360/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078258.153223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078258.153225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078258.153226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616879360:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078258.153229:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616879488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078258.153238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078258.153243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078258/real 1769078258] req@ffff995b84640e00 x1855007616879488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078258.153250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616879488/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078258.153255:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078258.153257:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078258.153258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616879488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078258.856574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078258.856639:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078259.176558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078259.176574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078259.176580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078259.176601:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078259.880560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078259.880621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078260.200584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078260.200585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078260.200619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078260.200621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078260.904571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078260.904663:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078261.224617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078261.224618:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078261.224619:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078261.224804:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078261.928577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078261.928646:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078262.248571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078262.248571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078262.248580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078262.248597:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078262.952554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078262.952614:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078263.272564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078263.272564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078263.272565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078263.272565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769078263.272600:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078263.272607:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078263.272611:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078263.272621:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5248 00000100:00080000:1.0:1769078263.272633:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078263.272638:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078263.272682:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616879616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078263.272701:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078263.272705:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078263.272706:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078263.272710:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5248 00000100:00080000:1.0:1769078263.272714:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078263.272718:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078263.272727:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078263.272735:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616879616:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078263.272739:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616879744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078263.272748:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078263.272750:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078263.272751:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078263.272755:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5248 00000100:00080000:1.0:1769078263.272758:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078263.272761:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078263.272764:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078263.272772:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616879872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078263.272779:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078263.272781:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078263.272783:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078263.272786:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5248 00000100:00080000:1.0:1769078263.272788:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078263.272792:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078263.272796:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078263.272805:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616880000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078263.272813:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078263.272816:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078263.272817:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078263.272820:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5248 00000100:00080000:1.0:1769078263.272823:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078263.272827:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078263.272831:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078263.272839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616879744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078263.272841:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616880128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078263.272850:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078263.272852:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078263.272854:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078263.272858:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5248 00000100:00080000:1.0:1769078263.272861:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078263.272865:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078263.272869:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078263.272869:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616879872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078263.272878:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616880256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078263.272886:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078263.272887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880000:192.168.204.132@tcp:8:kworker.0 00000100:00080000:1.0:1769078263.272890:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078263.272892:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078263.272895:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5248 00000100:00080000:1.0:1769078263.272899:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078263.272902:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078263.272907:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078263.272907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078263.272915:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616880384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078263.272925:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616880256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078263.272949:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616880384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078263.976543:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078263.976601:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078264.296566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078264.296566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078264.296609:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078264.296636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078265.000514:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078265.000582:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078265.320536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078265.320541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078265.320662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078265.320711:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078266.024547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078266.024596:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078266.344541:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078266.344541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078266.344541:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078266.344704:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:1.0:1769078266.344766:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769078266.344810:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769078266.344828:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078266.344864:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078266.344872:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078266.344879:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078266.344888:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078266.344898:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078266.344906:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769078266.344917:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078263/real 1769078266] req@ffff995b868c7480 x1855007616880128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078266.344943:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616880128/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078266.344962:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078266.344967:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078266.344972:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078266.344981:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078263/real 1769078266] req@ffff995b868c7100 x1855007616880256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078266.345018:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616880256/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078266.345024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078266.345026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078266.345029:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616880256:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078266.345035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078263/real 1769078266] req@ffff995b868c5500 x1855007616879744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078266.345047:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616879744/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078266.345057:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078266.345061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078266.345064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616879744:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078266.345070:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078263/real 1769078266] req@ffff995b868c4700 x1855007616879872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078266.345081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616879872/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078266.345092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078266.345097:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078266.345100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616879872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078266.345108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078263/real 1769078266] req@ffff995b868c7800 x1855007616880000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078266.345118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616880000/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078266.345126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078266.345130:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078266.345132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880000:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078266.345139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078263/real 1769078266] req@ffff995b868c5c00 x1855007616879616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078266.345150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616879616/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078266.345157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078266.345160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078266.345163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616879616:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078266.345193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078263/real 1769078266] req@ffff995b868c6a00 x1855007616880384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078266.345206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616880384/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078266.345212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078266.345216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078266.345218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616880384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078267.048552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078267.048598:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078267.368573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078267.368573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078267.368573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078267.368607:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078268.072565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078268.072684:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078268.392554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078268.392562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078268.392605:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078268.392611:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078268.392615:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078268.392628:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5254 00000100:00080000:2.0:1769078268.392641:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078268.392649:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078268.392700:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616880512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078268.392710:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078268.392726:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078268.392732:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078268.392734:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078268.392740:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5254 00000100:00080000:2.0:1769078268.392745:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078268.392752:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078268.392753:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616880512:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769078268.392773:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078268.392789:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616880640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078268.392802:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078268.392805:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078268.392807:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078268.392811:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5254 00000100:00100000:1.0:1769078268.392864:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078268.393049:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078268.393056:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078268.393062:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078268.393077:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616880768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078268.393087:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078268.393090:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078268.393092:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078268.393096:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5254 00000100:00080000:2.0:1769078268.393099:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078268.393104:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078268.393110:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1769078268.393112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078268.393156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078268/real 1769078268] req@ffff995b85ac8380 x1855007616880512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.393177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616880512/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.393188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.393193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078268.393197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616880512:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078268.393224:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078268.393248:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078268.393261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078268/real 1769078268] req@ffff995b85ac9880 x1855007616880640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.393274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616880640/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.393282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.393285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078268.393289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880640:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078268.393296:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078268.393313:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078268.393324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078268/real 1769078268] req@ffff995b85acb100 x1855007616880768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.393336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616880768/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.393352:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.393356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078268.393359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078268.393482:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616880896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078268.393504:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078268.393507:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078268.393509:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078268.393514:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5254 00000100:00080000:2.0:1769078268.393519:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078268.393524:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078268.393529:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078268.393544:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616881024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078268.393553:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078268.393558:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078268.393560:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078268.393564:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5254 00000100:00080000:2.0:1769078268.393568:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078268.393572:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078268.393577:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078268.393586:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616881152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078268.393594:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078268.393597:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078268.393599:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078268.393603:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5254 00000100:00080000:2.0:1769078268.393606:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078268.393609:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078268.393614:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078268.393623:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616881280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078268.395231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880896:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078268.395799:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078268.395826:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078268/real 1769078268] req@ffff995b85aca300 x1855007616880896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.395852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616880896/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.395866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.396023:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078268.396028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616880896:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078268.396035:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881024:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078268.396059:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078268.396071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078268/real 1769078268] req@ffff995b85acb800 x1855007616881024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.396081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616881024/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.396089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.396092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078268.396095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078268.396101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616881152:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078268.396119:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078268.396129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078268/real 1769078268] req@ffff995b85ac8a80 x1855007616881152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.396139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616881152/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.396146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.396149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078268.396152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616881152:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078268.396157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616881280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078268.396175:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078268.396185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078268/real 1769078268] req@ffff995b85ac9c00 x1855007616881280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.396195:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616881280/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.396201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.396203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078268.396205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616881280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078269.096541:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078269.096599:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078269.416544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078269.416553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078269.416572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078269.416674:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078270.120562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078270.120626:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078270.440548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078270.440548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078270.440557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078270.440585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078271.144545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078271.144614:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078271.464555:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078271.464555:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078271.464557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078271.464592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078272.168503:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078272.168592:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078272.488551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078272.488551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078272.488551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078272.488580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078273.192555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078273.192610:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078273.512570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078273.512570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078273.512570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078273.512580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078273.512609:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078273.512616:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078273.512620:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078273.512629:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5259 00000100:00080000:0.0:1769078273.512771:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078273.512783:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078273.512827:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616881408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078273.512851:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078273.512855:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078273.512857:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078273.512861:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5259 00000100:00080000:0.0:1769078273.512865:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078273.512868:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078273.512901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616881408:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769078273.512902:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078273.512917:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616881536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078273.512929:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078273.512932:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078273.512934:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078273.512939:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5259 00000100:00080000:0.0:1769078273.512943:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078273.512946:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078273.512951:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078273.512960:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616881664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078273.512978:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769078273.512982:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078273.512983:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078273.512987:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078273.512992:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5259 00000100:00080000:0.0:1769078273.512996:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078273.513000:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078273.513005:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078273.513018:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616881792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078273.513027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078273/real 1769078273] req@ffff995b84641500 x1855007616881408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078273.513029:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078273.513036:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078273.513040:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078273.513048:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5259 00000100:00080000:0.0:1769078273.513052:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078273.513056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616881408/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078273.513059:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078273.513066:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769078273.513072:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769078273.513080:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616881920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078273.513080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078273.513087:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616881408:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078273.513098:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078273.513102:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078273.513103:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078273.513108:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5259 00000100:00080000:0.0:1769078273.513111:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078273.513120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078273.513144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078273.513156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078273/real 1769078273] req@ffff995b84641180 x1855007616881536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078273.513168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616881536/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078273.513175:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769078273.513177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078273.513182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:0.0:1769078273.513183:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078273.513187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078273.513195:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078273.513200:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616882048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078273.513212:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769078273.513216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078273.513217:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078273.513221:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078273.513226:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5259 00000100:00100000:3.0:1769078273.513230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078273/real 1769078273] req@ffff995b84643800 x1855007616881664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078273.513232:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078273.513240:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078273.513246:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078273.513249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616881664/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078273.513258:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616882176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078273.513259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078273.513266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078273.513273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078273.513281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881792:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078273.513297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078273.513303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078273/real 1769078273] req@ffff995b84643480 x1855007616881792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078273.513310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616881792/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078273.513314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078273.513316:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078273.513318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881792:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078273.513321:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078273.513329:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078273.513338:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078273/real 1769078273] req@ffff995b84642d80 x1855007616881920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078273.513357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616881920/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078273.513367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078273.513369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078273.513373:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616881920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078273.513390:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616882048:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078273.513407:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078273.513417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078273/real 1769078273] req@ffff995b84643100 x1855007616882048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078273.513427:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616882048/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078273.513435:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078273.513438:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078273.513441:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616882048:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078273.513447:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616882176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078273.513485:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078273.513497:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078273/real 1769078273] req@ffff995b84643b80 x1855007616882176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078273.513595:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616882176/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078273.513603:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078273.513606:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078273.513609:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616882176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078274.216563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078274.216714:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078274.536527:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078274.536566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078274.536576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078274.536593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078275.240532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078275.240585:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078275.560548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078275.560557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078275.561531:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078275.561558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078276.264720:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078276.264894:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078276.584558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078276.584654:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078276.584687:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078276.586221:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078277.288546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078277.288611:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078277.608577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078277.608711:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078277.608755:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078277.609439:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769078278.312635:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078278.312687:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078278.632642:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078278.632653:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078278.632658:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078278.632676:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5264 00000100:00080000:1.0:1769078278.632688:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078278.632696:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078278.632737:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616882304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078278.632776:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078278.632780:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078278.632782:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078278.632803:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5264 00000100:00080000:1.0:1769078278.632807:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078278.632811:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078278.632821:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078278.632832:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616882432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078278.632841:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078278.632843:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078278.632845:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078278.632849:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5264 00000100:00080000:1.0:1769078278.632858:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078278.632863:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078278.632868:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078278.632878:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616882560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078278.632886:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078278.632889:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078278.632891:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078278.632894:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5264 00000100:00080000:1.0:1769078278.632897:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078278.632899:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078278.632903:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078278.632911:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616882688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078278.632925:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078278.632928:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078278.632929:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078278.632932:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5264 00000100:00080000:1.0:1769078278.632934:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078278.632937:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078278.632941:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078278.632949:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616882816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078278.632960:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078278.632963:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078278.632965:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078278.632968:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5264 00000100:00080000:1.0:1769078278.632971:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078278.632975:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078278.632979:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078278.632983:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616882304:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078278.632989:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616882944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078278.633000:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078278.633003:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078278.633005:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078278.633008:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5264 00000100:00080000:1.0:1769078278.633011:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078278.633015:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078278.633019:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078278.633028:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616883072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078278.633052:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078278.633092:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078278/real 1769078278] req@ffff995b868c6300 x1855007616882304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078278.633112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616882304/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078278.633122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078278.633126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078278.633130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616882304:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078278.633136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616882432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078278.633156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078278.633168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078278/real 1769078278] req@ffff995b868c6d80 x1855007616882432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078278.633194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616882432/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078278.633202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078278.633205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078278.633208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616882432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078278.633214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616882560:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078278.633232:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078278.633242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078278/real 1769078278] req@ffff995b868c4a80 x1855007616882560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078278.633260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616882560/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078278.633269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078278.633272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078278.633276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616882560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078278.633281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616882688:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078278.633299:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078278.633308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078278/real 1769078278] req@ffff995b868c6a00 x1855007616882688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078278.633318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616882688/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078278.633326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078278.633329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078278.633331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616882688:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078278.633336:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616882816:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078278.633352:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078278.633362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078278/real 1769078278] req@ffff995b868c5c00 x1855007616882816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078278.633370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616882816/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078278.633384:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078278.633387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078278.633390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616882816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078278.633417:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616882944:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078278.633433:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078278.633442:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078278/real 1769078278] req@ffff995b868c7800 x1855007616882944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078278.633481:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616882944/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078278.633491:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078278.633494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078278.633497:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616882944:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078278.633503:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616883072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078278.633521:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078278.633532:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078278/real 1769078278] req@ffff995b868c4700 x1855007616883072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078278.633543:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616883072/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078278.633564:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078278.633687:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078278.633694:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616883072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078278.633922:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078278.634051:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078278.634079:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078278.634080:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078279.336566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078279.336709:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078279.656530:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078279.656535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078279.656691:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078279.656719:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078280.360551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078280.360612:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078280.680594:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078280.680602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078280.680626:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078280.680727:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078281.384548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078281.384632:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078281.704600:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078281.704608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078281.704615:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078281.704623:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:1.0:1769078282.408609:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078282.408644:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078282.728596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078282.728596:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078282.728619:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078282.728702:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078283.432729:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078283.432767:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078283.752737:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769078283.752793:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078283.752798:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078283.752800:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078283.752807:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078283.752818:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5269 00000100:00080000:0.0:1769078283.752831:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078283.752839:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078283.752875:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616883200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078283.752895:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078283.752899:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078283.752900:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078283.752903:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5269 00000100:00080000:0.0:1769078283.752907:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078283.752910:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078283.752918:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078283.752930:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616883328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078283.752937:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078283.752940:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078283.752940:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078283.752943:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5269 00000100:00080000:0.0:1769078283.752945:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078283.752948:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078283.752951:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078283.752960:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616883456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078283.752966:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078283.752968:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078283.752969:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078283.752973:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5269 00000100:00080000:0.0:1769078283.752975:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078283.752978:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078283.752981:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078283.752990:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616883584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078283.752995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616883200:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078283.753002:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078283.753006:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078283.753008:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078283.753013:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5269 00000100:00080000:0.0:1769078283.753017:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078283.753021:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078283.753032:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078283.753046:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616883712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078283.753057:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078283.753061:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078283.753063:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078283.753067:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5269 00000100:00080000:0.0:1769078283.753072:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769078283.753072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078283.753078:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078283.753082:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078283.753091:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616883840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078283.753096:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078283.753098:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078283.753099:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078283.753101:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5269 00000100:00080000:0.0:1769078283.753102:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078283.753104:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078283.753106:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078283.753111:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616883968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078283.753179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078283/real 1769078283] req@ffff995b84642a00 x1855007616883200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078283.753197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616883200/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078283.753207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078283.753212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078283.753216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616883200:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078283.753226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616883328:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078283.753246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078283.753330:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078283.753376:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078283.755862:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078283/real 1769078283] req@ffff995b84640700 x1855007616883328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078283.755883:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616883328/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078283.755892:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078283.755896:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078283.755901:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616883328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078283.755909:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616883456:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078283.755938:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078283.755953:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078283/real 1769078283] req@ffff995b84641880 x1855007616883456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078283.755967:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616883456/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078283.755977:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078283.755981:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078283.755984:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616883456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078283.756012:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616883584:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078283.756029:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078283.756038:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078283/real 1769078283] req@ffff995b84643b80 x1855007616883584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078283.756050:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616883584/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078283.756059:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078283.756062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078283.756065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616883584:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078283.756071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616883712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078283.756084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078283.756093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078283/real 1769078283] req@ffff995b84643100 x1855007616883712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078283.756103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616883712/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078283.756112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078283.756115:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078283.756118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616883712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078283.756124:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616883840:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078283.756137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078283.756145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078283/real 1769078283] req@ffff995b84642d80 x1855007616883840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078283.756156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616883840/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078283.756170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078283.756173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078283.756176:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616883840:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078283.756182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616883968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078283.756195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078283.756204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078283/real 1769078283] req@ffff995b84643480 x1855007616883968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078283.756214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616883968/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078283.756223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078283.756226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078283.756229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616883968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078284.456658:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078284.456952:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078284.776619:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078284.776619:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078284.776767:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078284.776991:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078285.480573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078285.480642:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078285.800564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078285.800564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078285.800564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078285.800565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078286.504549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078286.504600:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078286.824564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078286.824565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078286.824565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078286.824569:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078287.528679:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078287.528752:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078287.848551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078287.848551:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078287.848551:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078287.848551:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078288.552573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078288.552654:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078288.872568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078288.872568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078288.872592:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078288.872602:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078288.872607:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078288.872611:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078288.872636:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5274 00000100:00100000:2.0:1769078288.872638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078288.872650:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078288.872659:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078288.872711:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616884096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078288.872744:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078288.872751:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078288.872754:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078288.872758:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5274 00000100:00080000:0.0:1769078288.872765:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078288.872777:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078288.872787:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078288.872794:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616884096:192.168.204.132@tcp:250:kworker.0 00000100:00100000:0.0:1769078288.872803:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616884224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078288.872812:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078288.872815:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078288.872816:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078288.872820:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5274 00000100:00080000:0.0:1769078288.872823:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078288.872827:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078288.872830:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078288.872836:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616884352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078288.872843:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078288.872845:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078288.872846:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078288.872849:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5274 00000100:00080000:0.0:1769078288.872851:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078288.872853:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078288.872860:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078288.872869:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616884480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078288.872875:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078288.872877:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078288.872879:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078288.872885:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5274 00000800:00000100:3.0:1769078288.872888:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078288.872894:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078288.872900:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078288.872914:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078288.872931:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616884608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078288.872942:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078288.872948:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1769078288.872949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078288/real 1769078288] req@ffff995b84643800 x1855007616884096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078288.872951:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078288.872969:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5274 00000100:00080000:0.0:1769078288.872975:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078288.872979:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078288.872985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616884096/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078288.872988:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078288.873004:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616884736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078288.873008:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078288.873021:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078288.873025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078288.873030:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078288.873034:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078288.873040:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616884096:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078288.873043:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5274 00000100:00080000:0.0:1769078288.873049:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078288.873055:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078288.873059:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078288.873067:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616884864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078288.873086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616884224:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078288.873120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078288.873146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078288/real 1769078288] req@ffff995b84641180 x1855007616884224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078288.873159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616884224/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078288.873167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078288.873175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078288.873178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616884224:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078288.873185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616884352:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078288.873202:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078288.873212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078288/real 1769078288] req@ffff995b84641500 x1855007616884352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078288.873223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616884352/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078288.873233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078288.873237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078288.873240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616884352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078288.873247:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616884480:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078288.873270:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078288.873280:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078288/real 1769078288] req@ffff995b84640e00 x1855007616884480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078288.873291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616884480/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078288.873301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078288.873304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078288.873307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616884480:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078288.873313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616884608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078288.873330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078288.873340:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078288/real 1769078288] req@ffff995b84642680 x1855007616884608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078288.873350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616884608/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078288.873358:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078288.873361:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078288.873364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616884608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078288.873370:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616884736:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078288.873386:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078288.873396:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078288/real 1769078288] req@ffff995b84640000 x1855007616884736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078288.873408:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616884736/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078288.873415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078288.873418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078288.873422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616884736:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078288.873428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616884864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078288.873444:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078288.873487:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078288/real 1769078288] req@ffff995b84640380 x1855007616884864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078288.873503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616884864/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078288.873512:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078288.873515:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078288.873518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616884864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078289.576596:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078289.576899:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078289.896563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078289.896563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078289.896563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078289.896612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078290.600604:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078290.600674:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078290.920576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078290.920577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078290.920577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078290.920595:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078291.624611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078291.624670:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078291.944522:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078291.944527:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078291.944536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078291.944536:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078292.648546:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078292.648627:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078292.968590:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078292.968590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078292.968590:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078292.968590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078293.672581:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078293.672749:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078293.992558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078293.992558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078293.992635:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078293.992673:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078293.992720:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078293.992727:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078293.992731:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078293.992740:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5279 00000100:00080000:1.0:1769078293.992750:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078293.992758:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078293.992793:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616884992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078293.992816:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078293.992820:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078293.992821:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078293.992826:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5279 00000100:00080000:1.0:1769078293.992829:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078293.992834:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078293.992843:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078293.992847:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616884992:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078293.992856:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616885120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078293.992867:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078293.992870:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078293.992871:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078293.992875:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5279 00000100:00080000:1.0:1769078293.992878:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078293.992881:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078293.992886:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078293.992894:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616885248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078293.992903:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078293.992906:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078293.992907:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078293.992912:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5279 00000800:00000100:3.0:1769078293.992917:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078293.992918:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078293.992923:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078293.992928:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078293.992937:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616885376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078293.992946:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078293.992949:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078293.992951:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078293.992955:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5279 00000100:00080000:1.0:1769078293.992958:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078293.992962:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078293/real 1769078293] req@ffff995b868c5500 x1855007616884992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078293.992964:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078293.992969:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078293.992981:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616885504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078293.992990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616884992/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078293.992991:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078293.992998:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078293.993001:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078293.993009:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5279 00000100:00080000:1.0:1769078293.993014:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078293.993016:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078293.993018:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078293.993023:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769078293.993025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078293.993033:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616885632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078293.993034:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616884992:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078293.993049:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078293.993053:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078293.993054:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078293.993058:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5279 00000100:00080000:1.0:1769078293.993062:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078293.993066:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616885120:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078293.993067:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078293.993074:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078293.993084:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616885760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078293.993098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078293.993115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078293/real 1769078293] req@ffff995b868c7100 x1855007616885120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078293.993129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616885120/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078293.993138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078293.993141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078293.993144:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616885120:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078293.993150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616885248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078293.993168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078293.993177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078293/real 1769078293] req@ffff995b868c7480 x1855007616885248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078293.993189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616885248/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078293.993205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078293.993208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078293.993211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616885248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078293.993233:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616885376:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078293.993253:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078293.993264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078293/real 1769078293] req@ffff995b868c4700 x1855007616885376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078293.993275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616885376/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078293.993284:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078293.993296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078293.993299:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616885376:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078293.993305:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616885504:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078293.993323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078293.993333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078293/real 1769078293] req@ffff995b868c7800 x1855007616885504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078293.993344:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616885504/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078293.993353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078293.993356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078293.993359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616885504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078293.993365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616885632:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078293.993394:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078293.993404:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078293/real 1769078293] req@ffff995b868c5c00 x1855007616885632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078293.993415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616885632/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078293.993423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078293.993426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078293.993429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616885632:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078293.993444:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616885760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078293.993477:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078293.993487:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078293/real 1769078293] req@ffff995b868c6a00 x1855007616885760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078293.993498:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616885760/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078293.993506:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078293.993509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078293.993512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616885760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078294.696528:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078294.696592:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078295.016590:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078295.016590:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078295.016596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078295.016638:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078295.720565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078295.720624:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078296.040538:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078296.040538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078296.040560:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078296.040591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078296.744507:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078296.744588:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078297.064559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078297.064559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078297.064562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078297.064564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078297.768544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078297.768603:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078298.088568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078298.088568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078298.088569:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078298.088610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078298.792530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078298.792603:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078299.112550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078299.112552:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078299.112553:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078299.112560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078299.112562:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078299.112569:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078299.112580:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5284 00000100:00080000:2.0:1769078299.112591:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078299.112598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769078299.112599:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078299.112640:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616885888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078299.112660:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078299.112664:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078299.112665:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078299.112668:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5284 00000100:00080000:2.0:1769078299.112671:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078299.112674:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078299.112682:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078299.112690:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616886016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078299.112694:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616885888:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078299.112701:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078299.112704:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078299.112705:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078299.112707:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5284 00000100:00080000:2.0:1769078299.112709:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078299.112711:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078299.112713:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078299.112722:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616886144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078299.112727:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078299.112737:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078299.112738:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078299.112741:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5284 00000100:00080000:2.0:1769078299.112746:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078299.112748:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078299.112752:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078299.112758:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616886272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078299.112763:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078299.112766:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078299.112768:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769078299.112770:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078299.112772:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5284 00000100:00080000:2.0:1769078299.112775:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078299.112777:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078299.112780:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078299.112787:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616886400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078299.112792:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078299.112794:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078299.112796:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078299.112798:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5284 00000100:00080000:2.0:1769078299.112800:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078299.112802:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078299.112805:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078299.112813:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078299/real 1769078299] req@ffff995b85acbb80 x1855007616885888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078299.112814:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616886528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078299.112836:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078299.112838:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078299.112840:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078299.112844:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5284 00000100:00100000:3.0:1769078299.112846:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616885888/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078299.112847:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078299.112851:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078299.112855:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078299.112865:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616886656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078299.112866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078299.112876:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078299.112883:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616885888:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078299.112907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886016:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078299.112929:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078299.112941:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078299/real 1769078299] req@ffff995b85acaa00 x1855007616886016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078299.112954:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616886016/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078299.112963:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078299.112966:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078299.112970:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886016:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078299.112975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078299.112992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078299.113001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078299/real 1769078299] req@ffff995b85ac9c00 x1855007616886144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078299.113012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616886144/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078299.113023:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078299.113026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078299.113029:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078299.113035:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886272:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078299.113051:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078299.113061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078299/real 1769078299] req@ffff995b85ac8a80 x1855007616886272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078299.113071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616886272/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078299.113080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078299.113083:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078299.113086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886272:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078299.113091:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078299.113107:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078299.113117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078299/real 1769078299] req@ffff995b85acb800 x1855007616886400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078299.113134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616886400/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078299.113142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078299.113144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078299.113147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078299.113152:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616886528:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078299.113169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078299.113179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078299/real 1769078299] req@ffff995b85aca300 x1855007616886528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078299.113190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616886528/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078299.113197:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078299.113207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078299.113210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616886528:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078299.113215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616886656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078299.113233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078299.113243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078299/real 1769078299] req@ffff995b85acb100 x1855007616886656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078299.113255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616886656/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078299.113267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078299.113270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078299.113273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616886656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078299.816589:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078299.816657:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078300.136601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078300.136608:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078300.136611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078300.136637:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078300.840542:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078300.840616:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078301.160573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078301.160581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078301.160598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078301.160607:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078301.864555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078301.864604:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078302.184549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078302.184549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078302.184554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078302.184559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078302.888562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078302.888621:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078303.208557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078303.208557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078303.208566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078303.208599:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078303.912564:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078303.912618:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078304.232567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078304.232567:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078304.232567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078304.232580:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078304.232586:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078304.232590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078304.232598:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5289 00000100:00100000:3.0:1769078304.232605:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078304.232610:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078304.232619:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078304.232660:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616886784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078304.232689:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078304.232692:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078304.232693:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078304.232696:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5289 00000100:00080000:1.0:1769078304.232699:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078304.232702:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078304.232710:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078304.232721:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616886912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078304.232721:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616886784:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078304.232733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078304.232735:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078304.232741:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078304.232744:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5289 00000100:00080000:1.0:1769078304.232747:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078304.232749:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078304.232753:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078304.232762:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616887040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078304.232767:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078304.232770:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078304.232771:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078304.232774:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5289 00000100:00080000:1.0:1769078304.232776:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078304.232778:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769078304.232782:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769078304.232783:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078304.232792:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616887168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078304.232798:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078304.232800:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078304.232801:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078304.232803:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5289 00000100:00080000:1.0:1769078304.232805:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078304.232807:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078304.232810:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078304.232817:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078304/real 1769078304] req@ffff995b868c4a80 x1855007616886784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078304.232819:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616887296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078304.232841:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078304.232845:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078304.232846:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078304.232849:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5289 00000100:00080000:1.0:1769078304.232851:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078304.232852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616886784/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078304.232856:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078304.232862:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769078304.232866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769078304.232870:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616887424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078304.232871:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078304.232878:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616886784:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078304.232890:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078304.232892:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078304.232893:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078304.232895:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5289 00000100:00080000:1.0:1769078304.232897:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078304.232899:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078304.232902:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078304.232903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078304.232910:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616887552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078304.232924:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078304.232939:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078304/real 1769078304] req@ffff995b868c6d80 x1855007616886912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078304.232957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616886912/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078304.232966:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078304.232969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078304.232973:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616886912:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078304.232979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887040:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078304.232996:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078304.233004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078304/real 1769078304] req@ffff995b868c6300 x1855007616887040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078304.233011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616887040/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078304.233018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078304.233020:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078304.233022:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078304.233025:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887168:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078304.233035:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078304.233040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078304/real 1769078304] req@ffff995b868c6a00 x1855007616887168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078304.233050:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616887168/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078304.233077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078304.233081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078304.233084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887168:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078304.233089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887296:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078304.233108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078304.233118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078304/real 1769078304] req@ffff995b868c5c00 x1855007616887296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078304.233130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616887296/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078304.233139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078304.233142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078304.233145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078304.233150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616887424:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078304.233164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078304.233171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078304/real 1769078304] req@ffff995b868c7800 x1855007616887424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078304.233178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616887424/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078304.233183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078304.233184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078304.233187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616887424:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078304.233198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616887552:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078304.233207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078304.233213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078304/real 1769078304] req@ffff995b868c4700 x1855007616887552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078304.233219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616887552/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078304.233223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078304.233225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078304.233226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616887552:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769078304.936634:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078304.936706:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078305.256560:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078305.256571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078305.256596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078305.258195:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078305.960537:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078305.960626:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078306.280546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078306.280546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078306.280563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078306.280574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078306.984568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078306.984613:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078307.304533:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078307.304533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078307.304546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078307.304551:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078308.008615:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078308.008639:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078308.328562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078308.328562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078308.328568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078308.328573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078309.032555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078309.032621:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078309.352514:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078309.352523:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078309.352529:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078309.352632:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078309.352638:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078309.352642:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078309.352651:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5295 00000100:00080000:1.0:1769078309.352661:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078309.352668:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078309.352743:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616887680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078309.352768:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078309.352772:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078309.352774:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078309.352778:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5295 00000100:00080000:1.0:1769078309.352789:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078309.352799:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616887680:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078309.352799:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078309.352810:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078309.352824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616887808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078309.352835:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078309.352838:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078309.352840:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078309.352844:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5295 00000800:00000100:0.0:1769078309.352869:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078309.352889:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078309.352931:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078309.352937:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078309.352943:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078309.352956:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616887936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078309.352966:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078309.352969:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078309.352971:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078309.352977:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078309/real 1769078309] req@ffff995b868c7480 x1855007616887680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078309.352983:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5295 00000100:00080000:1.0:1769078309.352989:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078309.352994:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078309.352999:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616887680/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078309.353002:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769078309.353011:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078309.353015:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078309.353021:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616887680:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078309.353022:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616888064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078309.353046:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078309.353050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887808:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078309.353051:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078309.353056:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078309.353062:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5295 00000100:00080000:1.0:1769078309.353066:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078309.353070:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078309.353075:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769078309.353081:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078309.353089:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616888192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078309.353102:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078309.353105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078309/real 1769078309] req@ffff995b868c7100 x1855007616887808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078309.353106:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078309.353109:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078309.353117:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5295 00000100:00080000:1.0:1769078309.353123:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078309.353128:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078309.353131:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616887808/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078309.353134:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078309.353145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078309.353150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078309.353151:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616888320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078309.353154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887808:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078309.353164:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887936:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078309.353170:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078309.353174:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078309.353177:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078309.353182:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5295 00000100:00080000:1.0:1769078309.353186:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078309.353191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078309.353194:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078309.353201:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078309.353208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078309/real 1769078309] req@ffff995b868c5500 x1855007616887936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078309.353216:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616888448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078309.353226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616887936/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078309.353239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078309.353243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078309.353246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616887936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078309.353252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888064:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078309.353300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078309.353312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078309/real 1769078309] req@ffff995b868c4700 x1855007616888064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078309.353321:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616888064/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078309.353326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078309.353328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078309.353330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888064:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078309.353342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078309.353355:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078309.353361:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078309/real 1769078309] req@ffff995b868c7800 x1855007616888192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078309.353367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616888192/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078309.353373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078309.353375:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078309.353376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078309.353379:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616888320:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078309.353387:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078309.353392:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078309/real 1769078309] req@ffff995b868c5c00 x1855007616888320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078309.353411:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616888320/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078309.353418:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078309.353420:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078309.353423:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616888320:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078309.353428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616888448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078309.353446:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078309.353471:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078309/real 1769078309] req@ffff995b868c6a00 x1855007616888448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078309.353479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616888448/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078309.353484:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078309.353486:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078309.353488:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616888448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078310.056527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078310.056583:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078310.376547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078310.376547:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078310.376556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078310.376581:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078311.080559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078311.080638:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078311.400531:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078311.400558:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078311.400578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078311.400684:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078312.104532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078312.104587:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078312.424577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078312.424578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078312.424602:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078312.424603:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078313.128550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078313.128609:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078313.448550:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078313.448550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078313.448550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078313.448550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078314.152524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078314.152573:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078314.472547:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078314.472547:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078314.472547:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078314.472553:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769078314.472581:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078314.472585:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078314.472588:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078314.472595:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5300 00000100:00080000:0.0:1769078314.472603:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078314.472646:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078314.472678:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616888576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078314.472698:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078314.472700:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078314.472701:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078314.472704:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5300 00000100:00080000:0.0:1769078314.472706:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078314.472709:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078314.472715:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078314.472723:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616888704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078314.472731:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078314.472731:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616888576:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078314.472738:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078314.472740:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078314.472743:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5300 00000100:00080000:0.0:1769078314.472745:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078314.472747:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078314.472750:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078314.472757:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616888832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078314.472767:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078314.472769:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078314.472770:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078314.472772:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5300 00000100:00080000:0.0:1769078314.472780:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078314.472783:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769078314.472786:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769078314.472788:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078314.472799:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616888960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078314.472805:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078314.472807:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078314.472808:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078314.472814:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5300 00000100:00080000:0.0:1769078314.472817:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078314.472820:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078314/real 1769078314] req@ffff995b84641c00 x1855007616888576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078314.472823:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078314.472829:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078314.472841:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616889088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078314.472848:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616888576/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078314.472849:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078314.472857:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078314.472860:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078314.472865:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5300 00000100:00080000:3.0:1769078314.472866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078314.472870:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078314.472873:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:0.0:1769078314.472877:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078314.472881:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616888576:192.168.204.132@tcp:250:kworker.0 00000008:00100000:0.0:1769078314.472882:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078314.472892:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616889216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078314.472897:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078314.472900:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078314.472902:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078314.472902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888704:192.168.204.132@tcp:38:kworker.0 00000100:00080000:0.0:1769078314.472910:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5300 00000100:00080000:0.0:1769078314.472914:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078314.472918:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:3.0:1769078314.472923:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:0.0:1769078314.472925:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078314.472932:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078314/real 1769078314] req@ffff995b84641f80 x1855007616888704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078314.472941:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616889344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078314.472942:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616888704/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078314.472967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078314.472972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078314.472976:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888704:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078314.472981:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888832:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078314.472992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078314.472999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078314/real 1769078314] req@ffff995b84640a80 x1855007616888832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078314.473005:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616888832/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078314.473014:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078314.473016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078314.473018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078314.473021:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888960:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078314.473030:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078314.473034:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078314/real 1769078314] req@ffff995b84640380 x1855007616888960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078314.473041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616888960/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078314.473046:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078314.473047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078314.473049:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616888960:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078314.473052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078314.473060:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078314.473065:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078314/real 1769078314] req@ffff995b84640000 x1855007616889088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078314.473071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616889088/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078314.473076:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078314.473077:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078314.473079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078314.473082:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616889216:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078314.473090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078314.473095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078314/real 1769078314] req@ffff995b84642680 x1855007616889216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078314.473100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616889216/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078314.473104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078314.473106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078314.473108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616889216:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078314.473119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616889344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078314.473127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078314.473132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078314/real 1769078314] req@ffff995b84640e00 x1855007616889344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078314.473138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616889344/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078314.473142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078314.473144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078314.473145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616889344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078315.176565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078315.176632:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078315.496553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078315.496553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078315.496557:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078315.496559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078316.200565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078316.200618:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078316.520557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078316.520557:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078316.520558:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078316.520558:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078317.224593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078317.224666:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078317.544587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078317.544587:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078317.544593:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078317.544613:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078318.248570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078318.248641:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078318.568546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078318.568546:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078318.568546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078318.568549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769078319.272676:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078319.272708:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078319.592582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078319.592582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078319.592625:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078319.592631:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078319.592635:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078319.592638:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078319.592645:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5305 00000100:00080000:3.0:1769078319.592657:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078319.592663:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078319.592679:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078319.592701:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616889472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078319.592724:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078319.592728:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078319.592731:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078319.592735:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5305 00000100:00080000:3.0:1769078319.592739:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078319.592748:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616889472:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078319.592748:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078319.592762:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078319.592778:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616889600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078319.592790:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078319.592793:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078319.592794:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078319.592799:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5305 00000100:00080000:3.0:1769078319.592803:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078319.592809:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078319.592812:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078319.592820:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078319.592834:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616889728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078319.592844:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078319.592848:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769078319.592851:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078319/real 1769078319] req@ffff995b84558000 x1855007616889472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078319.592865:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616889472/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078319.592872:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078319.592875:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078319.592877:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616889472:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078319.592899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889600:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078319.592916:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078319.592927:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078319.592931:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078319/real 1769078319] req@ffff995b84559880 x1855007616889600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078319.592940:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616889600/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078319.592947:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078319.592949:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078319.592952:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889600:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078319.592964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078319.592979:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078319.592989:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078319/real 1769078319] req@ffff995b84558700 x1855007616889728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078319.592998:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616889728/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078319.593016:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078319.593022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078319.593027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889728:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078319.593070:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5305 00000100:00080000:3.0:1769078319.593123:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078319.593130:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078319.593138:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078319.593156:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616889856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078319.593179:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078319.593184:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078319.593186:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078319.593191:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5305 00000100:00080000:3.0:1769078319.593196:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078319.593201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889856:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769078319.593202:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078319.593211:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078319.593225:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616889984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078319.593233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078319.593241:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078319.593245:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769078319.593248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078319/real 1769078319] req@ffff995b8455b480 x1855007616889856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078319.593249:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078319.593258:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5305 00000100:00080000:3.0:1769078319.593265:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078319.593266:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616889856/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078319.593273:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:1.0:1769078319.593278:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769078319.593280:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769078319.593283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078319.593287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889856:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078319.593295:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616890112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078319.593300:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889984:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769078319.593314:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078319.593317:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1769078319.593321:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078319.593322:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078319.593329:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5305 00000100:00100000:1.0:1769078319.593331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078319/real 1769078319] req@ffff995b8455ad80 x1855007616889984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078319.593335:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078319.593344:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078319.593346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616889984/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078319.593350:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769078319.593354:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078319.593357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078319.593361:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616889984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078319.593365:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616890240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078319.593372:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616890112:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078319.593398:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078319.593408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078319/real 1769078319] req@ffff995b8455a300 x1855007616890112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078319.593415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616890112/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078319.593420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078319.593422:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078319.593424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616890112:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078319.593429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616890240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078319.593439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078319.593444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078319/real 1769078319] req@ffff995b84559c00 x1855007616890240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078319.593451:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616890240/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078319.593472:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078319.593478:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078319.593484:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616890240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078320.296557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078320.296613:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078320.616534:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078320.616540:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078320.616546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078320.616565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078321.320547:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078321.320620:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078321.640564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078321.640564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078321.640564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078321.640564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078322.344559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078322.344616:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078322.664572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078322.664572:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078322.664572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078322.664572:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078323.368554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078323.368621:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078323.688568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078323.688568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078323.688568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078323.688568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078324.392527:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078324.392583:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078324.712566:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078324.712566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078324.712578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078324.712583:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078324.712607:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078324.712613:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078324.712617:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078324.712630:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5310 00000100:00080000:3.0:1769078324.712642:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078324.712649:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078324.712703:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616890368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078324.712726:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078324.712729:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078324.712730:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078324.712732:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5310 00000100:00080000:3.0:1769078324.712734:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078324.712737:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078324.712745:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078324.712754:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616890496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078324.712762:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078324.712765:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078324.712766:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078324.712769:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5310 00000100:00080000:3.0:1769078324.712771:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078324.712777:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616890368:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078324.712778:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078324.712787:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078324.712795:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616890624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078324.712806:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078324.712808:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078324.712809:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078324.712812:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5310 00000100:00080000:3.0:1769078324.712814:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078324.712816:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078324.712819:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078324.712826:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616890752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078324.712831:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078324.712834:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078324.712836:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078324.712838:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5310 00000100:00080000:3.0:1769078324.712841:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078324.712843:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078324.712846:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078324.712852:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616890880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078324.712857:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078324.712859:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078324.712862:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078324.712870:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078324.712871:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5310 00000100:00080000:3.0:1769078324.712877:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078324.712882:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078324.712885:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078324.712895:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616891008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078324.712902:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078324.712905:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078324.712906:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078324.712910:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5310 00000100:00080000:3.0:1769078324.712912:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078324.712916:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078324.712920:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078324.712932:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616891136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078324.712935:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078324/real 1769078324] req@ffff995b84559c00 x1855007616890368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078324.712965:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616890368/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078324.712975:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078324.712980:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078324.712984:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616890368:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078324.713008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616890496:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078324.713043:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078324.713055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078324/real 1769078324] req@ffff995b8455a300 x1855007616890496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078324.713066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616890496/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078324.713074:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078324.713077:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078324.713081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616890496:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078324.713087:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616890624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078324.713105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078324.713115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078324/real 1769078324] req@ffff995b8455ad80 x1855007616890624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078324.713127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616890624/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078324.713136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078324.713138:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078324.713141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616890624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078324.713147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616890752:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078324.713166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078324.713175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078324/real 1769078324] req@ffff995b8455b480 x1855007616890752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078324.713186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616890752/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078324.713194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078324.713198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078324.713201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616890752:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078324.713207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616890880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078324.713224:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078324.713234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078324/real 1769078324] req@ffff995b84558700 x1855007616890880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078324.713245:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616890880/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078324.713253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078324.713256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078324.713258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616890880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078324.713264:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616891008:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078324.713281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078324.713290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078324/real 1769078324] req@ffff995b84559880 x1855007616891008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078324.713300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616891008/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078324.713308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078324.713321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078324.713324:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616891008:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078324.713330:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616891136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078324.713346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078324.713356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078324/real 1769078324] req@ffff995b84558000 x1855007616891136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078324.713367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616891136/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078324.713376:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078324.713379:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078324.713382:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616891136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078325.416583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078325.416672:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078325.736556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078325.736556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078325.736598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078325.736611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078326.440549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078326.440605:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078326.760521:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078326.760535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078326.760569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078326.760575:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078327.464545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078327.464607:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078327.784544:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078327.784544:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078327.784544:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078327.784563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078328.488588:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078328.488658:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078328.808581:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078328.808588:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078328.808588:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078328.808588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078329.512553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078329.512569:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078329.832602:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078329.832608:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078329.832610:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078329.832611:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769078329.832663:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078329.832669:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078329.832673:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078329.832684:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5315 00000100:00080000:1.0:1769078329.832697:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078329.832704:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078329.832745:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616891264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078329.832768:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078329.832771:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078329.832772:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078329.832775:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5315 00000100:00080000:1.0:1769078329.832777:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078329.832780:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078329.832790:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078329.832801:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616891392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078329.832804:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616891264:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078329.832824:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078329.832826:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078329.832827:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078329.832831:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5315 00000100:00080000:1.0:1769078329.832833:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078329.832837:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078329.832840:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078329.832848:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616891520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078329.832853:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078329.832856:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078329.832858:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078329.832860:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5315 00000100:00080000:1.0:1769078329.832862:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078329.832865:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078329.832869:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078329.832880:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616891648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078329.832885:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078329.832888:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078329.832890:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078329.832892:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5315 00000100:00080000:1.0:1769078329.832896:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078329.832898:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078329.832901:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078329.832909:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616891776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078329.832917:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078329.832920:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078329.832922:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078329.832939:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616891392:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078329.832940:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5315 00000100:00080000:1.0:1769078329.832946:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078329.832951:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078329.832957:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078329.832966:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616891904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078329.832973:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078329.832976:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078329.832978:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078329.832982:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5315 00000100:00080000:1.0:1769078329.832985:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078329.832990:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078329.832992:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616891520:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769078329.832995:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078329.833007:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616892032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078329.833031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616891648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078329.833063:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616891776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078329.833090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616891904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078329.833109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616892032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078330.536557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078330.536658:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078330.856517:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078330.856518:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078330.856518:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078330.856568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078331.560521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078331.560693:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078331.880548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078331.880548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078331.880561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078331.880600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078332.584548:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078332.584608:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078332.904558:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078332.904561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078332.904562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078332.904706:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000400:00000100:1.0:1769078332.904774:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:1.0:1769078332.904811:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769078332.904829:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078332.904886:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078332.904895:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078332.904903:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078332.904909:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078332.904915:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:1.0:1769078332.904922:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769078332.905599:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078329/real 1769078332] req@ffff995b868c7800 x1855007616891904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078332.905630:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616891904/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078332.905642:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078332.905647:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078332.905651:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616891904:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078332.905657:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078329/real 1769078332] req@ffff995b868c4700 x1855007616892032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078332.905667:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616892032/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078332.905675:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078332.905678:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078332.905682:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616892032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078332.905689:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078329/real 1769078332] req@ffff995b868c6d80 x1855007616891392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078332.905701:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616891392/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078332.905708:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078332.905712:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078332.905715:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616891392:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078332.905721:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078329/real 1769078332] req@ffff995b868c4a80 x1855007616891520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078332.905732:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616891520/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078332.905741:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078332.905744:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078332.905747:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616891520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078332.905754:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078329/real 1769078332] req@ffff995b868c6a00 x1855007616891648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078332.905765:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616891648/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078332.905773:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078332.905778:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078332.905781:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616891648:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078332.905788:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078329/real 1769078332] req@ffff995b868c5c00 x1855007616891776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078332.905798:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616891776/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078332.905806:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078332.905809:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078332.905811:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616891776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078332.905817:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078329/real 1769078332] req@ffff995b868c6300 x1855007616891264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078332.905826:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616891264/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078332.905832:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078332.905835:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078332.905838:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616891264:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078333.608583:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078333.608664:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078333.928582:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078333.928584:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078333.928609:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078333.928656:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078334.632590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078334.632672:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078334.952616:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078334.952616:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078334.952616:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078334.952658:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078334.952673:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078334.952682:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078334.952686:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078334.952695:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5320 00000100:00080000:2.0:1769078334.952705:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078334.952710:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078334.952739:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616892160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078334.952762:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078334.952764:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078334.952765:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078334.952768:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5320 00000100:00080000:2.0:1769078334.952770:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078334.952773:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078334.952781:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078334.952789:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616892288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078334.952798:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078334.952806:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769078334.952807:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616892160:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078334.952809:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078334.952822:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5320 00000100:00080000:2.0:1769078334.952826:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078334.952828:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078334.952831:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078334.952867:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616892416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078334.952874:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078334.952877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078334.952878:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078334.952881:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5320 00000100:00080000:2.0:1769078334.952884:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078334.952887:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078334.952891:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078334.952901:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616892544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078334.952907:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078334.952910:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078334.952911:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078334.952914:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5320 00000100:00080000:2.0:1769078334.952935:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078334.952941:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078334.952946:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078334.952959:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616892672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078334.952968:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078334.952981:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078334.952986:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078334.952989:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078334.952996:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5320 00000100:00080000:2.0:1769078334.953002:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078334.953007:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078334.953012:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078334.953027:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616892800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078334.953037:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078334.953043:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078334.953046:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078334.953050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078334/real 1769078334] req@ffff995b85ac9880 x1855007616892160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078334.953057:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5320 00000100:00080000:2.0:1769078334.953072:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078334.953077:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078334.953083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616892160/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769078334.953084:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769078334.953108:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769078334.953109:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616892928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078334.953124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078334.953137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616892160:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078334.953187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616892288:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078334.953223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078334.953248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078334/real 1769078334] req@ffff995b85ac8380 x1855007616892288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078334.953269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616892288/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078334.953276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078334.953280:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078334.953283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616892288:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078334.953291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616892416:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078334.953312:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078334.953323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078334/real 1769078334] req@ffff995b85acb100 x1855007616892416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078334.953335:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616892416/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078334.953346:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078334.953349:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078334.953353:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616892416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078334.953358:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616892544:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078334.953381:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078334.953392:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078334/real 1769078334] req@ffff995b85aca300 x1855007616892544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078334.953404:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616892544/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078334.953414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078334.953417:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078334.953420:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616892544:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078334.953427:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616892672:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078334.953448:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078334.953496:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078334/real 1769078334] req@ffff995b85acb800 x1855007616892672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078334.953508:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616892672/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078334.953519:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078334.953522:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078334.953526:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616892672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078334.953612:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616892800:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078334.953638:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078334.953650:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078334/real 1769078334] req@ffff995b85ac8a80 x1855007616892800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078334.953668:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616892800/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078334.953675:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078334.953678:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078334.953682:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616892800:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078334.953688:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616892928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078334.953706:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078334.953734:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078334/real 1769078334] req@ffff995b85ac9c00 x1855007616892928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078334.953748:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616892928/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078334.953757:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078334.953760:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078334.953763:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616892928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078335.656519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078335.656642:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078335.976601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078335.976601:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078335.976649:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078335.976726:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078336.680561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078336.680626:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078337.000592:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078337.000592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078337.000592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078337.000592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078337.704561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078337.704724:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078338.024555:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078338.024556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078338.024559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078338.024578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078338.728560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078338.728630:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078339.048548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078339.048549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078339.048556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078339.048574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078339.752569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078339.752634:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078340.072542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078340.072543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078340.072548:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078340.072548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078340.072580:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078340.072585:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078340.072589:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078340.072598:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5325 00000100:00080000:2.0:1769078340.072608:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078340.072614:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078340.072647:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616893056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078340.072671:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078340.072674:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078340.072676:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078340.072680:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5325 00000100:00080000:2.0:1769078340.072684:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078340.072688:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078340.072696:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616893056:192.168.204.132@tcp:250:kworker.0 00000008:00100000:2.0:1769078340.072697:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078340.072710:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616893184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078340.072721:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078340.072724:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078340.072726:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078340.072739:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5325 00000100:00080000:2.0:1769078340.072745:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769078340.072749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078340.072765:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078340.072772:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078340.072785:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616893312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078340.072795:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078340.072799:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078340.072802:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078340.072802:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078340/real 1769078340] req@ffff995b85acaa00 x1855007616893056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078340.072810:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5325 00000100:00080000:2.0:1769078340.072816:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078340.072823:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078340.072828:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078340.072832:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616893056/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078340.072840:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616893440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078340.072842:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078340.072849:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769078340.072856:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078340.072857:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616893056:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078340.072862:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078340.072866:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078340.072870:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5325 00000100:00080000:2.0:1769078340.072874:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078340.072877:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078340.072882:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078340.072886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616893184:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078340.072893:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616893568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078340.072903:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078340.072907:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078340.072908:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078340.072914:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5325 00000800:00000100:3.0:1769078340.072915:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078340.072920:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078340.072925:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078340.072929:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078340.072937:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078340/real 1769078340] req@ffff995b85acbb80 x1855007616893184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078340.072940:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616893696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078340.072960:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078340.072961:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616893184/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078340.072967:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078340.072971:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078340.072973:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078340.072978:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5325 00000100:00080000:3.0:1769078340.072979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769078340.072984:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078340.072984:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616893184:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078340.072991:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078340.072993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616893312:192.168.204.132@tcp:38:kworker.0 00000008:00100000:2.0:1769078340.072998:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078340.073011:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616893824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078340.073014:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078340.073024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078340/real 1769078340] req@ffff995b85ac9c00 x1855007616893312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078340.073046:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616893312/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078340.073060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078340.073064:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078340.073067:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616893312:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078340.073073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616893440:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078340.073091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078340.073102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078340/real 1769078340] req@ffff995b85ac8a80 x1855007616893440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078340.073110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616893440/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078340.073115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078340.073117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078340.073119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616893440:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078340.073130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616893568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078340.073142:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078340.073148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078340/real 1769078340] req@ffff995b85acb800 x1855007616893568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078340.073154:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616893568/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078340.073159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078340.073161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078340.073163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616893568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078340.073166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616893696:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078340.073174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078340.073179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078340/real 1769078340] req@ffff995b85aca300 x1855007616893696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078340.073185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616893696/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078340.073190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078340.073191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078340.073193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616893696:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078340.073196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616893824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078340.073203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078340.073208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078340/real 1769078340] req@ffff995b85acb100 x1855007616893824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078340.073214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616893824/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078340.073218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078340.073220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078340.073221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616893824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078340.776557:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078340.776631:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078341.096549:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078341.096550:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078341.096550:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078341.096550:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078341.800598:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078341.800657:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078342.120613:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078342.120618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078342.120618:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078342.120745:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078342.824585:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078342.824645:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078343.144533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078343.144546:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078343.144548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078343.144570:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078343.848628:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078343.848682:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078344.168537:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078344.168548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078344.168562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078344.168614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078344.872559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078344.872613:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078345.192531:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078345.192533:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078345.192543:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078345.192568:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078345.192573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078345.192574:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078345.192579:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078345.192588:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5330 00000100:00080000:2.0:1769078345.192599:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078345.192605:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078345.192635:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616893952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078345.192657:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078345.192660:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078345.192662:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078345.192682:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5330 00000100:00080000:2.0:1769078345.192686:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078345.192691:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078345.192699:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078345.192701:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616893952:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078345.192719:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616894080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078345.192730:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078345.192733:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078345.192735:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078345.192740:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5330 00000100:00080000:2.0:1769078345.192753:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078345.192757:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078345.192763:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769078345.192768:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078345.192778:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616894208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078345.192788:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078345.192791:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078345.192792:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078345.192795:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5330 00000100:00080000:2.0:1769078345.192798:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078345.192801:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078345.192805:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078345.192807:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078345/real 1769078345] req@ffff995b85ac8380 x1855007616893952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078345.192815:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616894336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078345.192825:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078345.192827:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078345.192829:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078345.192833:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616893952/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078345.192834:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5330 00000100:00080000:2.0:1769078345.192838:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078345.192843:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078345.192849:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1769078345.192854:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769078345.192858:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616894464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078345.192863:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078345.192870:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616893952:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078345.192873:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078345.192879:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078345.192882:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078345.192887:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5330 00000100:00080000:2.0:1769078345.192891:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078345.192896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894080:192.168.204.132@tcp:38:kworker.0 00000100:00080000:2.0:1769078345.192899:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078345.192907:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078345.192920:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616894592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078345.192926:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078345.192936:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078345.192941:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769078345.192942:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078345/real 1769078345] req@ffff995b85ac9880 x1855007616894080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078345.192944:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078345.192958:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616894080/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078345.192964:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5330 00000100:00080000:2.0:1769078345.192971:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078345.192973:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078345.192978:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:3.0:1769078345.192982:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000008:00100000:2.0:1769078345.192987:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078345.192987:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894080:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078345.192999:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078345.193007:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616894720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078345.193025:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078345.193038:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078345/real 1769078345] req@ffff995b85acb100 x1855007616894208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078345.193050:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616894208/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078345.193059:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078345.193062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078345.193065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894208:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078345.193071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894336:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078345.193100:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078345.193110:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078345/real 1769078345] req@ffff995b85aca300 x1855007616894336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078345.193121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616894336/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078345.193130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078345.193134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078345.193137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894336:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078345.193142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894464:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078345.193157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078345.193166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078345/real 1769078345] req@ffff995b85acb800 x1855007616894464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078345.193183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616894464/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078345.193192:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078345.193195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078345.193198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078345.193214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616894592:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078345.193231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078345.193241:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078345/real 1769078345] req@ffff995b85ac8a80 x1855007616894592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078345.193252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616894592/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078345.193260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078345.193263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078345.193266:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616894592:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078345.193272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616894720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078345.193288:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078345.193297:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078345/real 1769078345] req@ffff995b85ac9c00 x1855007616894720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078345.193308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616894720/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078345.193321:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078345.193324:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078345.193327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616894720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078345.896560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078345.896611:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078346.216528:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078346.216548:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078346.216552:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078346.216579:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078346.920551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078346.920597:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078347.240517:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078347.240529:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078347.240530:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078347.241490:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078347.944563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078347.944633:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078348.264528:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078348.264553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078348.264559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078348.264576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078348.968540:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078348.968596:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078349.288561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078349.288564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078349.288572:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078349.288589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078349.992567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078349.992756:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078350.312564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078350.312564:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078350.312570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078350.312592:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078350.312598:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078350.312602:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078350.312618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078350.312621:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5336 00000100:00080000:2.0:1769078350.312634:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078350.312641:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078350.312682:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616894848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078350.312707:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078350.312711:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078350.312712:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078350.312716:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5336 00000100:00080000:2.0:1769078350.312720:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078350.312723:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078350.312731:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078350.312742:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616894976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078350.312762:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078350.312765:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078350.312767:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078350.312770:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5336 00000100:00080000:2.0:1769078350.312773:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078350.312776:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078350.312780:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078350.312791:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616895104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078350.312798:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078350.312801:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078350.312803:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078350.312806:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5336 00000100:00080000:2.0:1769078350.312809:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078350.312812:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078350.312816:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078350.312824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616895232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078350.312832:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078350.312834:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078350.312835:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078350.312838:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5336 00000100:00080000:2.0:1769078350.312841:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078350.312844:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078350.312848:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078350.312856:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616895360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078350.312864:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078350.312867:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078350.312869:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078350.312872:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5336 00000100:00080000:2.0:1769078350.312875:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078350.312878:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078350.312882:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078350.312891:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616895488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078350.312898:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078350.312901:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078350.312902:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078350.312906:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5336 00000100:00080000:2.0:1769078350.312908:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078350.312911:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078350.312915:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078350.312924:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616895616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078350.312943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616894848:192.168.204.132@tcp:250:kworker.0 00000800:00000100:3.0:1769078350.312990:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078350.313018:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078350/real 1769078350] req@ffff995b85acbb80 x1855007616894848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078350.313029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616894848/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078350.313036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078350.313038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078350.313041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616894848:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078350.313045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894976:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078350.313066:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078350.313074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078350/real 1769078350] req@ffff995b85acaa00 x1855007616894976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078350.313085:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616894976/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078350.313090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078350.313092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078350.313094:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616894976:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078350.313098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616895104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078350.313108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078350.313113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078350/real 1769078350] req@ffff995b85ac9c00 x1855007616895104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078350.313119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616895104/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078350.313126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078350.313127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078350.313129:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616895104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078350.313132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616895232:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078350.313140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078350.313145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078350/real 1769078350] req@ffff995b85ac8a80 x1855007616895232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078350.313152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616895232/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078350.313167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078350.313169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078350.313172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616895232:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078350.313178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616895360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078350.313194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078350.313205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078350/real 1769078350] req@ffff995b85acb800 x1855007616895360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078350.313216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616895360/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078350.313225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078350.313228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078350.313230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616895360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078350.313234:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616895488:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078350.313247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078350.313253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078350/real 1769078350] req@ffff995b85aca300 x1855007616895488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078350.313260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616895488/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078350.313265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078350.313275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078350.313279:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616895488:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078350.313306:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616895616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078350.313332:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078350.313353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078350/real 1769078350] req@ffff995b85acb100 x1855007616895616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078350.313365:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616895616/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078350.313373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078350.313376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078350.313379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616895616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078351.016600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078351.016694:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078351.336600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078351.336601:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078351.336618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078351.336633:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078352.040562:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078352.040668:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078352.360641:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078352.360642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078352.360642:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078352.360642:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078353.064559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078353.064608:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078353.384577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078353.384577:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078353.384577:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078353.384577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078354.088574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078354.088636:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078354.408507:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078354.408544:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078354.408567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078354.408573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078355.112558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078355.112614:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078355.432519:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078355.432532:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078355.432566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078355.432598:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078355.432599:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078355.432605:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078355.432609:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078355.432617:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5341 00000100:00080000:3.0:1769078355.432628:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078355.432636:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078355.432668:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616895744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078355.432692:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078355.432695:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078355.432697:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078355.432701:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5341 00000100:00080000:3.0:1769078355.432704:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078355.432707:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078355.432713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616895744:192.168.204.132@tcp:250:kworker.0 00000008:00100000:3.0:1769078355.432714:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078355.432728:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616895872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078355.432735:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078355.432738:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078355.432740:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078355.432744:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5341 00000100:00080000:3.0:1769078355.432761:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078355.432766:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769078355.432771:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:3.0:1769078355.432773:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078355.432789:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616896000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078355.432798:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078355.432802:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078355.432804:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078355.432809:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5341 00000100:00100000:2.0:1769078355.432811:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078355/real 1769078355] req@ffff995b8455bb80 x1855007616895744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078355.432825:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616895744/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078355.432832:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078355.432836:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078355.432839:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616895744:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078355.432855:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616895872:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078355.432868:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078355.432876:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078355/real 1769078355] req@ffff995b84558a80 x1855007616895872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078355.432883:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616895872/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078355.432892:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078355.432895:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078355.432896:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616895872:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078355.432899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078355.432909:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078355.432914:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078355/real 1769078355] req@ffff995b8455a680 x1855007616896000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078355.432920:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616896000/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078355.432930:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078355.432932:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078355.432934:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896000:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078355.432942:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078355.433129:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078355.433203:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078355.433232:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616896128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078355.433249:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078355.433252:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078355.433254:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078355.433258:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5341 00000100:00100000:2.0:1769078355.433272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896128:192.168.204.132@tcp:8:kworker.0 00000100:00080000:3.0:1769078355.433277:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078355.433284:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769078355.433292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078355.433300:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078355/real 1769078355] req@ffff995b8455b100 x1855007616896128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078355.433305:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078355.433309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616896128/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078355.433315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078355.433317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078355.433319:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896128:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078355.433338:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616896256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078355.433379:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078355.433383:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078355.433385:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078355.433386:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078355.433406:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078355.433406:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5341 00000100:00100000:2.0:1769078355.433417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078355/real 1769078355] req@ffff995b8455aa00 x1855007616896256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078355.433425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616896256/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078355.433425:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078355.433435:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078355.433435:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769078355.433441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078355.433444:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896256:192.168.204.132@tcp:8:kworker.0 00000008:00100000:3.0:1769078355.433480:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078355.433529:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616896384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078355.433554:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078355.433568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616896384:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078355.433575:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078355.433578:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078355.433593:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5341 00000800:00000100:0.0:1769078355.433596:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078355.433600:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078355.433610:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078355/real 1769078355] req@ffff995b84559500 x1855007616896384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078355.433620:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078355.433621:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616896384/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078355.433630:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078355.433632:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078355.433639:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078355.433642:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616896384:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078355.433651:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616896512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078355.433673:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616896512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078355.433689:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078355.433699:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078355/real 1769078355] req@ffff995b8455aa00 x1855007616896512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078355.433708:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616896512/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078355.433715:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078355.433718:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078355.433722:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616896512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078356.136570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078356.136625:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078356.456591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078356.456614:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078356.456615:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078356.456648:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078357.160567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078357.160620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078357.480576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078357.480576:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078357.480576:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078357.480600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078358.184558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078358.184608:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078358.504570:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078358.504575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078358.504579:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078358.504624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078359.208571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078359.208629:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078359.528574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078359.528574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078359.528692:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078359.528911:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078360.232571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078360.232638:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078360.552578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078360.552598:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769078360.552693:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078360.552698:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078360.552703:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078360.552712:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5346 00000100:00080000:0.0:1769078360.552721:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078360.552728:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078360.552762:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616896640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078360.552787:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078360.552790:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078360.552792:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078360.552796:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5346 00000100:00080000:0.0:1769078360.552799:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078360.552804:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078360.552812:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078360.552824:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616896768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078360.552827:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616896640:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078360.552840:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078360.552843:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078360.552845:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078360.552850:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5346 00000100:00080000:0.0:1769078360.552853:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078360.552856:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078360.552860:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078360.552869:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616896896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078360.552876:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078360.552879:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078360.552880:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078360.552883:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5346 00000100:00080000:0.0:1769078360.552886:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078360.552889:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078360.552893:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078360.552900:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616897024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078360.552908:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078360.552910:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078360.552912:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078360.552915:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5346 00000100:00080000:0.0:1769078360.552918:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078360.552922:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078360.552925:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078360.552934:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616897152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078360.552942:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078360.552947:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1769078360.552948:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:0.0:1769078360.552962:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078360.552968:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5346 00000100:00080000:0.0:1769078360.552971:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078360.552974:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078360.552979:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078360.552990:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078360/real 1769078360] req@ffff995b84641500 x1855007616896640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078360.552991:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616897280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078360.553017:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078360.553021:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078360.553023:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078360.553027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616896640/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078360.553029:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5346 00000100:00080000:0.0:1769078360.553034:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078360.553041:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769078360.553046:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769078360.553047:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769078360.553054:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078360.553059:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616897408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078360.553061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616896640:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078360.553091:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896768:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078360.553116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078360.553129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078360/real 1769078360] req@ffff995b84641180 x1855007616896768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078360.553141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616896768/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078360.553150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078360.553153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078360.553156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896768:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078360.553162:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078360.553182:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078360.553193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078360/real 1769078360] req@ffff995b84643800 x1855007616896896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078360.553204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616896896/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078360.553220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078360.553223:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078360.553227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616896896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078360.553232:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897024:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078360.553247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078360.553254:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078360/real 1769078360] req@ffff995b84643480 x1855007616897024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078360.553261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616897024/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078360.553267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078360.553269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078360.553271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897024:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078360.553274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897152:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078360.553283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078360.553288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078360/real 1769078360] req@ffff995b84642d80 x1855007616897152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078360.553297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616897152/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078360.553305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078360.553308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078360.553311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078360.553316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616897280:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078360.553334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078360.553345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078360/real 1769078360] req@ffff995b84643100 x1855007616897280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078360.553356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616897280/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078360.553361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078360.553363:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078360.553365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616897280:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078360.553369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616897408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078360.553382:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078360.553432:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078360/real 1769078360] req@ffff995b84643b80 x1855007616897408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078360.553442:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616897408/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078360.553450:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078360.553481:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078360.553486:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616897408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078360.554494:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078360.554521:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078361.256571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078361.256672:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078361.576564:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078361.576622:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078361.577185:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078361.577208:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078362.280553:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078362.280633:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078362.600556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078362.600556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078362.600561:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078362.600587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078363.304810:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078363.305416:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078363.624573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078363.624579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078363.625479:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078363.625493:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078364.328570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078364.328624:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078364.648557:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078364.648563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078364.648564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078364.648582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078365.352558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078365.352629:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078365.672521:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078365.672525:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078365.672552:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078365.672647:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078365.672653:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078365.672656:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078365.672664:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5351 00000100:00080000:1.0:1769078365.672674:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078365.672679:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078365.672713:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616897536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078365.672737:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078365.672740:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078365.672742:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078365.672747:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5351 00000100:00080000:1.0:1769078365.672751:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078365.672755:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078365.672765:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078365.672774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616897536:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078365.672779:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616897664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078365.672793:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078365.672796:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078365.672798:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078365.672801:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5351 00000100:00080000:1.0:1769078365.672805:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078365.672808:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078365.672813:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078365.672822:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616897792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078365.672831:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078365.672836:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078365.672836:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078365.672840:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078365.672846:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5351 00000100:00080000:1.0:1769078365.672851:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078365.672855:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078365.672860:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078365.672871:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616897920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078365.672880:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078365/real 1769078365] req@ffff995b868c5500 x1855007616897536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078365.672881:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078365.672888:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078365.672891:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078365.672899:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5351 00000100:00080000:1.0:1769078365.672903:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078365.672906:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616897536/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078365.672910:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078365.672918:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1769078365.672923:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078365.672931:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078365.672931:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616898048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078365.672938:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616897536:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078365.672948:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078365.672952:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078365.672954:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078365.672958:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5351 00000100:00080000:1.0:1769078365.672962:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078365.672966:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078365.672968:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897664:192.168.204.132@tcp:38:kworker.0 00000008:00100000:1.0:1769078365.672973:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078365.672987:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616898176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078365.672997:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078365.672999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078365.673001:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078365.673005:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078365.673009:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5351 00000100:00080000:1.0:1769078365.673014:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078365.673017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078365/real 1769078365] req@ffff995b868c7100 x1855007616897664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078365.673020:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078365.673036:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616897664/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078365.673037:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078365.673050:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769078365.673053:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616898304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078365.673057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078365.673064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897664:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078365.673075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078365.673097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078365.673109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078365/real 1769078365] req@ffff995b868c7480 x1855007616897792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078365.673122:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078365.673125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616897792/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078365.673133:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078365.673136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078365.673138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078365.673143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897920:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078365.673155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078365.673161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078365/real 1769078365] req@ffff995b868c6300 x1855007616897920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078365.673168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616897920/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078365.673173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078365.673175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078365.673177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616897920:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078365.673179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078365.673187:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078365.673192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078365/real 1769078365] req@ffff995b868c5c00 x1855007616898048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078365.673198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616898048/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078365.673207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078365.673209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078365.673210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078365.673221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616898176:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078365.673229:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078365.673234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078365/real 1769078365] req@ffff995b868c6a00 x1855007616898176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078365.673240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616898176/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078365.673245:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078365.673247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078365.673248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616898176:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078365.673251:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616898304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078365.673259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078365.673264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078365/real 1769078365] req@ffff995b868c4a80 x1855007616898304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078365.673270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616898304/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078365.673274:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078365.673276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078365.673278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616898304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078366.376535:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078366.376587:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078366.696549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078366.696549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078366.696570:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078366.696593:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078367.400550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078367.400629:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078367.720561:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078367.720573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078367.720592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078367.720612:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078368.424567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078368.424674:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078368.744585:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078368.744585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078368.744591:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078368.744592:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078369.448558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078369.448628:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078369.768565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078369.768565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078369.768580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078369.768610:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078370.472545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078370.472596:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078370.792549:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078370.792549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078370.792549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078370.792554:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078370.792583:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078370.792589:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078370.792593:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078370.792601:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5356 00000100:00080000:3.0:1769078370.792610:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078370.792615:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078370.792644:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616898432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078370.792665:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078370.792668:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078370.792669:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078370.792672:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5356 00000100:00080000:3.0:1769078370.792674:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078370.792676:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078370.792683:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078370.792691:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616898560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078370.792698:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078370.792700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616898432:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078370.792702:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078370.792705:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078370.792708:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5356 00000100:00080000:3.0:1769078370.792710:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078370.792712:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078370.792715:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078370.792723:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616898688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078370.792733:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078370.792735:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078370.792736:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078370.792738:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5356 00000100:00080000:3.0:1769078370.792740:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078370.792742:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078370.792745:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769078370.792753:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078370.792754:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616898816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078370.792761:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078370.792763:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078370.792765:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078370.792767:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5356 00000100:00080000:3.0:1769078370.792770:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078370.792771:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078370.792774:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078370.792780:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616898944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078370.792785:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078370.792788:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078370/real 1769078370] req@ffff995b8455b100 x1855007616898432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078370.792788:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078370.792791:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078370.792797:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5356 00000100:00080000:3.0:1769078370.792802:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078370.792805:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078370.792807:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078370.792808:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616898432/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078370.792816:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616899072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078370.792818:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078370.792822:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078370.792825:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616898432:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078370.792835:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078370.792837:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078370.792838:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078370.792841:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5356 00000100:00080000:3.0:1769078370.792843:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078370.792845:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078370.792848:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898560:192.168.204.132@tcp:38:kworker.0 00000008:00100000:3.0:1769078370.792850:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078370.792861:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616899200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078370.792867:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078370.792878:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078370/real 1769078370] req@ffff995b8455a680 x1855007616898560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078370.792886:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616898560/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078370.792892:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078370.792894:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078370.792896:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898560:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078370.792899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898688:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078370.792908:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078370.792913:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078370/real 1769078370] req@ffff995b84558a80 x1855007616898688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078370.792919:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616898688/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078370.792925:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078370.792927:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078370.792928:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078370.792931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898816:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078370.792940:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078370.792945:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078370/real 1769078370] req@ffff995b8455bb80 x1855007616898816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078370.792951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616898816/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078370.792956:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078370.792957:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078370.792959:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898816:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078370.792962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898944:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078370.792969:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078370.792974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078370/real 1769078370] req@ffff995b84558000 x1855007616898944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078370.792980:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616898944/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078370.792985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078370.792986:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078370.792988:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616898944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078370.792990:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616899072:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078370.793003:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078370.793012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078370/real 1769078370] req@ffff995b84559880 x1855007616899072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078370.793022:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616899072/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078370.793029:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078370.793032:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078370.793035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616899072:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078370.793052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616899200:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078370.793069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078370.793080:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078370/real 1769078370] req@ffff995b84558700 x1855007616899200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078370.793091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616899200/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078370.793100:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078370.793103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078370.793106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616899200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078371.496567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078371.496623:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078371.816559:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078371.816559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078371.816574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078371.816661:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078372.520530:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078372.520579:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078372.840556:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078372.840556:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078372.840556:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078372.840556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078373.544574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078373.544653:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078373.864563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078373.864563:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078373.864563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078373.864669:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078374.568575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078374.568647:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078374.888565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078374.888565:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078374.888574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078374.888672:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078375.592565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078375.592627:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078375.912571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078375.912571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078375.912571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078375.912571:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078375.912583:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078375.912587:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078375.912597:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5361 00000100:00080000:3.0:1769078375.912608:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078375.912615:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078375.912621:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078375.912648:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616899328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078375.912674:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078375.912677:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078375.912679:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078375.912683:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5361 00000100:00080000:3.0:1769078375.912686:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078375.912689:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078375.912700:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078375.912706:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616899328:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078375.912713:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616899456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078375.912723:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078375.912726:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078375.912728:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078375.912732:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5361 00000100:00080000:3.0:1769078375.912735:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078375.912739:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078375.912743:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078375.912755:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616899584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078375.912763:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078375.912767:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078375.912769:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769078375.912774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078375.912777:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5361 00000100:00080000:3.0:1769078375.912784:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078375.912788:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078375.912794:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078375.912804:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616899712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078375.912813:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078375.912818:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078375/real 1769078375] req@ffff995b8455b480 x1855007616899328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078375.912818:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078375.912822:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078375.912829:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5361 00000100:00080000:3.0:1769078375.912835:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078375.912839:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078375.912842:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078375.912845:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616899328/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078375.912853:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616899840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078375.912857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078375.912860:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078375.912865:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:3.0:1769078375.912865:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078375.912866:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078375.912873:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5361 00000100:00100000:0.0:1769078375.912875:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616899328:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078375.912876:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078375.912881:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078375.912886:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078375.912895:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616899968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078375.912904:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616899456:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078375.912916:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078375.912921:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078375.912922:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078375.912927:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5361 00000100:00080000:3.0:1769078375.912932:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078375.912934:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078375.912939:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078375.912945:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078375.912950:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078375/real 1769078375] req@ffff995b8455ad80 x1855007616899456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078375.912959:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616900096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078375.912967:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616899456/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078375.912977:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078375.912981:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078375.912985:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616899456:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078375.912999:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616899584:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078375.913019:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078375.913034:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078375/real 1769078375] req@ffff995b8455a300 x1855007616899584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078375.913045:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616899584/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078375.913055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078375.913058:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078375.913061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616899584:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078375.913067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616899712:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078375.913081:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078375.913089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078375/real 1769078375] req@ffff995b84559c00 x1855007616899712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078375.913106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616899712/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078375.913115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078375.913118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078375.913121:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616899712:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078375.913126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616899840:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078375.913139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078375.913168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078375/real 1769078375] req@ffff995b84558700 x1855007616899840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078375.913180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616899840/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078375.913198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078375.913202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078375.913208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616899840:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078375.913225:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616899968:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078375.913247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078375.913257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078375/real 1769078375] req@ffff995b84559880 x1855007616899968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078375.913268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616899968/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078375.913288:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078375.913291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078375.913294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616899968:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078375.913299:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616900096:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078375.913313:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078375.913326:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078375/real 1769078375] req@ffff995b84558000 x1855007616900096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078375.913337:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616900096/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078375.913344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078375.913347:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078375.913350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616900096:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078376.616555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078376.616620:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078376.936568:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078376.936568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078376.936568:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078376.936602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078377.640571:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078377.640635:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078377.960578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078377.960609:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078377.960636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078377.960640:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078378.664623:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078378.665271:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078378.984534:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078378.984566:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078378.984567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078378.984589:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078379.688600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078379.688662:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078380.008580:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078380.008580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078380.008580:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078380.008608:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078380.712536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078380.712640:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078381.032574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078381.032586:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078381.032602:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078381.032608:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078381.032616:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078381.032622:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078381.032627:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078381.032637:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5366 00000100:00080000:2.0:1769078381.032648:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078381.032657:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078381.032705:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616900224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078381.032731:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078381.032734:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078381.032735:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078381.032738:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5366 00000100:00080000:2.0:1769078381.032742:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078381.032745:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078381.032754:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078381.032765:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616900224:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078381.032767:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616900352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078381.032781:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078381.032783:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078381.032784:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078381.032787:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5366 00000100:00080000:2.0:1769078381.032789:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078381.032792:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078381.032795:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078381.032809:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616900480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078381.032818:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078381.032822:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078381.032824:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078381.032828:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5366 00000100:00080000:2.0:1769078381.032833:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078381.032836:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078381.032841:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078381.032847:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078381.032860:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616900608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078381.032877:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078381.032881:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078381/real 1769078381] req@ffff995b85ac9880 x1855007616900224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078381.032881:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078381.032884:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078381.032891:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5366 00000100:00080000:2.0:1769078381.032898:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078381.032904:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078381.032909:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078381.032911:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616900224/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078381.032924:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616900736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078381.032925:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078381.032933:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078381.032940:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616900224:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078381.032944:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078381.032948:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078381.032949:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078381.032953:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5366 00000100:00080000:2.0:1769078381.032956:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078381.032959:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078381.032963:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078381.032966:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616900352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078381.032975:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616900864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078381.032983:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078381.032986:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1769078381.032989:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078381.032990:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078381.032997:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5366 00000100:00080000:2.0:1769078381.033000:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078381.033002:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078381.033005:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078381.033006:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078381/real 1769078381] req@ffff995b85ac8380 x1855007616900352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078381.033015:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616900992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078381.033025:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616900352/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078381.033037:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078381.033042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078381.033047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616900352:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078381.033054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616900480:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078381.033087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078381.033096:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078381/real 1769078381] req@ffff995b85acb100 x1855007616900480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078381.033103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616900480/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078381.033110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078381.033113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078381.033115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616900480:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078381.033119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616900608:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078381.033129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078381.033135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078381/real 1769078381] req@ffff995b85aca300 x1855007616900608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078381.033142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616900608/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078381.033147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078381.033149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078381.033150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616900608:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078381.033153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616900736:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078381.033162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078381.033167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078381/real 1769078381] req@ffff995b85acb800 x1855007616900736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078381.033174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616900736/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078381.033179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078381.033180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078381.033182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616900736:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078381.033197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616900864:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078381.033213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078381.033223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078381/real 1769078381] req@ffff995b85ac8a80 x1855007616900864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078381.033235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616900864/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078381.033244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078381.033247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078381.033250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616900864:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078381.033255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616900992:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078381.033273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078381.033284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078381/real 1769078381] req@ffff995b85ac9c00 x1855007616900992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078381.033297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616900992/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078381.033304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078381.033307:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078381.033310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616900992:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078381.736570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078381.736641:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078382.056661:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078382.056694:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078382.056721:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078382.056957:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078382.760570:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078382.760636:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078383.080536:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078383.080536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078383.080536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078383.080548:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078383.784575:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078383.785248:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078384.104573:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078384.104573:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078384.104751:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078384.104764:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078384.808590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078384.808653:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078385.128541:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078385.128542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078385.128553:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078385.128568:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078385.832539:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078385.832594:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078386.152571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078386.152571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:2.0:1769078386.152578:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078386.152585:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078386.152586:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078386.152590:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078386.152604:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5371 00000100:00080000:2.0:1769078386.152613:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078386.152618:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078386.152637:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078386.152653:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616901120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078386.152676:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078386.152678:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078386.152680:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078386.152682:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5371 00000100:00080000:2.0:1769078386.152684:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078386.152687:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078386.152693:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078386.152702:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616901248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078386.152710:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078386.152716:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616901120:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078386.152718:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078386.152723:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078386.152736:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5371 00000100:00080000:2.0:1769078386.152744:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078386.152747:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078386.152750:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078386.152768:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616901376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078386.152773:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078386.152775:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078386.152777:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078386.152779:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5371 00000100:00080000:2.0:1769078386.152781:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078386.152783:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078386.152787:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078386.152793:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616901504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078386.152800:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078386.152802:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078386.152803:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078386.152804:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5371 00000100:00080000:2.0:1769078386.152808:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078386.152810:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078386.152813:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1769078386.152815:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078386.152828:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616901632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078386.152836:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078386.152839:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078386.152841:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078386.152846:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5371 00000100:00080000:2.0:1769078386.152850:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078386.152854:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078386.152862:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078386.152868:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078386/real 1769078386] req@ffff995b85acaa00 x1855007616901120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078386.152877:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616901760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078386.152895:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078386.152900:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616901120/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078386.152901:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078386.152906:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078386.152924:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078386.152928:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5371 00000100:00080000:0.0:1769078386.152933:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:2.0:1769078386.152935:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078386.152942:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616901120:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078386.152947:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078386.152957:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078386.152969:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616901888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078386.152991:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616901248:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078386.153018:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078386.153033:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078386/real 1769078386] req@ffff995b85acbb80 x1855007616901248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078386.153041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616901248/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078386.153047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078386.153049:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078386.153051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616901248:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078386.153055:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616901376:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078386.153065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078386.153071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078386/real 1769078386] req@ffff995b85ac9c00 x1855007616901376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078386.153082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616901376/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078386.153089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078386.153092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078386.153095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616901376:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078386.153108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616901504:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078386.153128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078386.153139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078386/real 1769078386] req@ffff995b85ac8a80 x1855007616901504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078386.153150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616901504/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078386.153156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078386.153158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078386.153160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616901504:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078386.153164:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616901632:192.168.204.132@tcp:8:kworker.0 00000800:00000100:0.0:1769078386.153178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078386.153184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078386/real 1769078386] req@ffff995b85acb800 x1855007616901632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078386.153191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616901632/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078386.153200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078386.153202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078386.153204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616901632:192.168.204.132@tcp:8:kworker.0 00000100:00100000:0.0:1769078386.153207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616901760:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078386.153216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078386.153221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078386/real 1769078386] req@ffff995b85aca300 x1855007616901760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078386.153227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616901760/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078386.153233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078386.153235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078386.153236:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616901760:192.168.204.132@tcp:38:kworker.0 00000100:00100000:0.0:1769078386.153250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616901888:192.168.204.132@tcp:38:kworker.0 00000800:00000100:0.0:1769078386.153259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078386.153264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078386/real 1769078386] req@ffff995b85acb100 x1855007616901888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078386.153271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616901888/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078386.153275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078386.153277:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:0.0:1769078386.153279:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616901888:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078386.856550:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078386.856629:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078387.176553:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078387.176578:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078387.176578:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078387.176627:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078387.880517:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078387.880575:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078388.200543:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078388.200549:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078388.200567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078388.200580:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078388.904521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078388.904625:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078389.224531:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078389.224531:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078389.224536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078389.224538:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078389.928524:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078389.928605:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078390.248584:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078390.248592:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078390.249007:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078390.250005:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078390.952551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078390.952626:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078391.272562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078391.272562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078391.272562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:0.0:1769078391.272606:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078391.272612:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1769078391.272616:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078391.272617:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078391.272630:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5376 00000100:00080000:0.0:1769078391.272641:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078391.272649:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078391.272682:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641880 x1855007616902016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078391.272713:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078391.272718:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078391.272720:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078391.272725:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5376 00000100:00080000:0.0:1769078391.272730:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078391.272734:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078391.272744:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078391.272758:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640700 x1855007616902144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078391.272768:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078391.272772:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078391.272773:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078391.272777:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5376 00000100:00080000:0.0:1769078391.272780:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078391.272783:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078391.272787:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078391.272797:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642a00 x1855007616902272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078391.272804:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078391.272807:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078391.272809:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078391.272812:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5376 00000100:00080000:0.0:1769078391.272815:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078391.272819:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078391.272823:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078391.272845:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643b80 x1855007616902400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078391.272855:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078391.272857:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078391.272859:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078391.272862:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5376 00000100:00080000:0.0:1769078391.272866:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078391.272869:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078391.272873:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078391.272884:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643100 x1855007616902528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078391.272931:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078391.272934:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078391.272935:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078391.272940:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5376 00000100:00080000:0.0:1769078391.272943:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078391.272946:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078391.272951:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078391.272968:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642d80 x1855007616902656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078391.272979:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078391.272979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616902016:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078391.272986:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078391.272989:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078391.272993:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5376 00000100:00080000:0.0:1769078391.272996:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078391.273000:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078391.273005:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078391.273014:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643480 x1855007616902784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078391.273046:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078391.273079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078391/real 1769078391] req@ffff995b84641880 x1855007616902016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078391.273097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641880 x1855007616902016/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078391.273106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078391.273111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078391.273115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616902016:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078391.273123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616902144:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078391.273145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078391.273159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078391/real 1769078391] req@ffff995b84640700 x1855007616902144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078391.273171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640700 x1855007616902144/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078391.273185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078391.273188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078391.273193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616902144:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078391.273199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616902272:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078391.273221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078391.273233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078391/real 1769078391] req@ffff995b84642a00 x1855007616902272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078391.273246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642a00 x1855007616902272/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078391.273260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078391.273265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078391.273269:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616902272:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078391.273276:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616902400:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078391.273295:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078391.273305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078391/real 1769078391] req@ffff995b84643b80 x1855007616902400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078391.273316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643b80 x1855007616902400/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078391.273324:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078391.273326:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078391.273329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616902400:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078391.273335:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616902528:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078391.273351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078391.273361:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078391/real 1769078391] req@ffff995b84643100 x1855007616902528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078391.273372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643100 x1855007616902528/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078391.273380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078391.273383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078391.273386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616902528:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078391.273413:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616902656:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078391.273430:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078391.273440:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078391/real 1769078391] req@ffff995b84642d80 x1855007616902656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078391.273450:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642d80 x1855007616902656/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078391.273483:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078391.273487:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078391.273491:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616902656:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078391.273497:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616902784:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078391.273518:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078391.273529:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078391/real 1769078391] req@ffff995b84643480 x1855007616902784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078391.273540:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643480 x1855007616902784/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078391.273547:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078391.273550:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078391.273553:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616902784:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078391.976565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078391.976737:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078392.296579:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078392.296589:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078392.296600:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078392.296626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078393.000561:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078393.000629:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078393.320564:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078393.320565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078393.320565:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078393.320594:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078394.024544:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078394.024713:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078394.344567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078394.344576:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078394.344576:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078394.344660:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078395.048578:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078395.049118:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078395.368565:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078395.368565:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078395.368892:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078395.370176:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078396.072576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078396.072799:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078396.392559:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078396.392559:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078396.392648:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:3.0:1769078396.395490:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078396.395499:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078396.395504:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078396.395514:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5382 00000100:00100000:2.0:1769078396.395518:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:3.0:1769078396.395525:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078396.395533:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078396.395565:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616902912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078396.395589:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078396.395592:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078396.395594:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078396.395597:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5382 00000100:00080000:3.0:1769078396.395600:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078396.395603:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078396.395611:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078396.395619:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616903040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078396.395627:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078396.395630:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078396.395631:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078396.395634:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5382 00000100:00080000:3.0:1769078396.395636:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078396.395639:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078396.395642:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078396.395649:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616903168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078396.395656:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078396.395658:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078396.395659:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078396.395662:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5382 00000100:00080000:3.0:1769078396.395664:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078396.395667:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078396.395670:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078396.395676:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616903296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078396.395683:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078396.395685:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078396.395687:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078396.395689:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5382 00000100:00080000:3.0:1769078396.395692:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078396.395694:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078396.395697:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078396.395703:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616903424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078396.395710:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078396.395713:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078396.395715:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078396.395719:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5382 00000100:00080000:3.0:1769078396.395721:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078396.395724:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078396.395727:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078396.395730:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616902912:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078396.395737:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616903552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078396.395745:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078396.395747:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078396.395749:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078396.395752:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5382 00000100:00080000:3.0:1769078396.395754:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078396.395757:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078396.395760:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078396.395770:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616903680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078396.395850:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078396.395880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078396.395903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078396.395922:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903424:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078396.395946:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616903552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078396.395966:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616903680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078397.096565:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078397.096629:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078397.417229:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078397.417380:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078397.417428:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078397.417775:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078398.120577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078398.120605:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078398.440644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078398.440667:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078398.440847:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078398.440862:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078399.144567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078399.146109:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078399.464769:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078399.464828:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078399.464829:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078399.465081:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000400:00000100:0.0:1769078399.465135:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:0.0:1769078399.465166:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769078399.465179:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769078399.465203:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769078399.465209:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769078399.465218:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769078399.465227:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:0.0:1769078399.465236:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769078399.465242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078396/real 1769078399] req@ffff995b8455bb80 x1855007616902912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1769078399.465247:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:2.0:1769078399.465270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616902912/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078399.465280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078399.465284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078399.465288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616902912:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078399.465295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078396/real 1769078399] req@ffff995b84558a80 x1855007616903040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078399.465307:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616903040/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078399.465313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078399.465317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078399.465320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903040:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078399.465327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078396/real 1769078399] req@ffff995b8455a680 x1855007616903168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078399.465337:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616903168/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078399.465344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078399.465347:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078399.465350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903168:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078399.465356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078396/real 1769078399] req@ffff995b8455b100 x1855007616903296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078399.465364:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616903296/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078399.465372:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078399.465375:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078399.465378:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903296:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078399.465389:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078396/real 1769078399] req@ffff995b8455aa00 x1855007616903424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078399.465399:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616903424/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078399.465407:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078399.465410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078399.465413:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903424:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078399.465434:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078396/real 1769078399] req@ffff995b84559500 x1855007616903552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078399.465445:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616903552/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078399.465451:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078399.465476:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078399.465480:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616903552:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078399.465487:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078396/real 1769078399] req@ffff995b84558000 x1855007616903680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078399.465499:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616903680/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078399.465508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078399.465512:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078399.465514:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616903680:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078400.168532:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078400.170025:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078400.488614:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078400.488644:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078400.488782:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078400.488796:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078401.192560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078401.192753:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078401.512627:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078401.512680:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078401.513648:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078401.513657:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078401.513662:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078401.513674:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5387 00000100:00080000:2.0:1769078401.513689:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078401.513700:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078401.513776:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616903808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078401.513818:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078401.513822:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078401.513824:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078401.513829:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5387 00000100:00080000:2.0:1769078401.513833:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078401.513838:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078401.513852:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078401.513865:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616903936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078401.513876:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078401.513879:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078401.513881:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078401.513886:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5387 00000100:00080000:2.0:1769078401.513890:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078401.513893:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078401.513898:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078401.513912:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616904064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078401.513923:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078401.513926:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078401.513929:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078401.513933:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5387 00000100:00080000:2.0:1769078401.513936:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078401.513940:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078401.513945:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078401.513964:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616904192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078401.513973:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078401.513977:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078401.513979:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078401.513983:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5387 00000100:00080000:2.0:1769078401.513986:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078401.513990:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078401.513995:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078401.514005:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616904320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078401.514013:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078401.514016:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078401.514018:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078401.514022:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5387 00000100:00080000:2.0:1769078401.514025:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078401.514029:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078401.514034:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078401.514044:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616904448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078401.514053:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078401.514056:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078401.514058:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078401.514062:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5387 00000100:00080000:2.0:1769078401.514065:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078401.514069:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078401.514073:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078401.514082:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616904576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078401.515766:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078401.515880:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078401.515967:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616903808:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769078401.516065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078401.516133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078401/real 1769078401] req@ffff995b85ac8380 x1855007616903808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078401.516154:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616903808/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078401.516165:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078401.516170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078401.516174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616903808:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078401.516183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903936:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078401.516206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078401.516220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078401/real 1769078401] req@ffff995b85ac9880 x1855007616903936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078401.516229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616903936/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078401.516235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078401.516238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078401.516240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616903936:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078401.516244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904064:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078401.516259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078401.516265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078401/real 1769078401] req@ffff995b85acb100 x1855007616904064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078401.516272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616904064/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078401.516283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078401.516285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078401.516287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904064:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078401.516290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904192:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078401.516301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078401.516309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078401/real 1769078401] req@ffff995b85aca300 x1855007616904192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078401.516318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616904192/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078401.516326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078401.516329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078401.516333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904192:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078401.516339:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904320:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078401.516357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078401.516367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078401/real 1769078401] req@ffff995b85acb800 x1855007616904320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078401.516378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616904320/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078401.516393:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078401.516395:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078401.516398:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904320:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078401.516402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616904448:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078401.516418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078401.516426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078401/real 1769078401] req@ffff995b85ac8a80 x1855007616904448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078401.516438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616904448/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078401.516446:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078401.516449:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078401.516452:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616904448:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078401.516501:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616904576:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078401.516520:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078401.516530:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078401/real 1769078401] req@ffff995b85ac9c00 x1855007616904576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078401.516541:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616904576/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078401.516548:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078401.516551:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078401.516554:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616904576:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078402.216593:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078402.216666:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078402.536537:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078402.536612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078402.536636:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078402.536737:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078403.240586:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078403.240948:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078403.560622:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078403.560622:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078403.560638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078403.560819:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078404.264560:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078404.266733:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078404.584571:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078404.584571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078404.584571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078404.584690:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078405.288551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078405.294472:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078405.608597:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078405.608723:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078405.609154:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078405.609176:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078406.312620:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078406.312681:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078406.632599:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078406.632602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078406.632607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078406.632614:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078406.632621:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078406.632645:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5392 00000100:00080000:1.0:1769078406.632660:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078406.632670:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078406.632676:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078406.632725:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616904704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078406.632748:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769078406.632758:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078406.632764:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078406.632767:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078406.632772:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5392 00000100:00080000:1.0:1769078406.632784:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078406.632793:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078406.632798:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616904704:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769078406.632806:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078406.632822:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616904832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078406.632833:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078406.632837:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078406.632840:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078406.632844:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5392 00000100:00080000:1.0:1769078406.632849:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078406.632852:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078406.632858:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078406.632869:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616904960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078406.632878:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078406.632883:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1769078406.632885:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078406.632886:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078406.632893:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5392 00000100:00080000:1.0:1769078406.632897:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078406.632902:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078406.632907:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078406.632919:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616905088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078406.632928:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078406.632932:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078406.632934:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078406.632938:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5392 00000100:00100000:2.0:1769078406.632942:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078406/real 1769078406] req@ffff995b868c6d80 x1855007616904704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078406.632945:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078406.632952:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078406.632960:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078406.632977:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616905216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078406.632984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616904704/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078406.632993:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078406.632997:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078406.632998:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078406.633002:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078406.633005:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769078406.633013:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5392 00000100:00080000:1.0:1769078406.633017:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769078406.633020:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616904704:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078406.633024:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078406.633031:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078406.633044:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616905344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078406.633048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904832:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078406.633064:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078406.633068:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078406.633071:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769078406.633075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078406.633077:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5392 00000100:00080000:1.0:1769078406.633084:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078406.633089:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078406.633092:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078406/real 1769078406] req@ffff995b868c4700 x1855007616904832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078406.633096:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078406.633103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616904832/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078406.633110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769078406.633113:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616905472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078406.633115:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078406.633121:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904832:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078406.633140:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904960:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078406.633163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078406.633174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078406/real 1769078406] req@ffff995b868c7800 x1855007616904960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078406.633187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616904960/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078406.633203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078406.633207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078406.633211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616904960:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078406.633216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905088:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078406.633237:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078406.633248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078406/real 1769078406] req@ffff995b868c4a80 x1855007616905088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078406.633258:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616905088/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078406.633264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078406.633267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078406.633269:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905088:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078406.633272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905216:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078406.633286:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078406.633292:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078406/real 1769078406] req@ffff995b868c6a00 x1855007616905216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078406.633298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616905216/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078406.633304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078406.633306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078406.633309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905216:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078406.633320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616905344:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078406.633330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078406.633336:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078406/real 1769078406] req@ffff995b868c5c00 x1855007616905344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078406.633343:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616905344/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078406.633347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078406.633349:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078406.633351:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616905344:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078406.633354:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616905472:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078406.633362:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078406.633368:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078406/real 1769078406] req@ffff995b868c6300 x1855007616905472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078406.633374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616905472/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078406.633379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078406.633381:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078406.633382:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616905472:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769078407.336679:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078407.336694:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078407.656634:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078407.656634:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078407.656835:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078407.657016:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078408.360624:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078408.361675:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078408.680556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078408.680650:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078408.680788:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078408.680790:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078409.384602:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078409.384669:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078409.704535:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078409.704682:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078409.704717:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078409.706164:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078410.408613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078410.408645:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078410.728653:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078410.728953:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078410.729019:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078410.729252:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769078411.432585:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078411.432600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078411.752569:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078411.752571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078411.752681:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078411.752810:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:2.0:1769078411.753327:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078411.753338:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078411.753353:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078411.753365:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5397 00000100:00080000:2.0:1769078411.753379:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078411.753387:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078411.753430:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616905600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078411.753497:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078411.753502:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078411.753504:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078411.753511:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616905600:192.168.204.132@tcp:250:kworker.0 00000100:00080000:2.0:1769078411.753515:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5397 00000100:00080000:2.0:1769078411.753525:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078411.753533:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078411.753554:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078411.753579:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616905728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078411.753597:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078411.753600:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078411.753603:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078411.753606:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5397 00000100:00080000:2.0:1769078411.753610:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078411.753614:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078411.753619:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078411.753631:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616905856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078411.753640:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078411.753643:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078411.753645:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078411.753648:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5397 00000100:00080000:2.0:1769078411.753653:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078411.753656:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078411.753661:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078411.753671:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616905984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078411.753679:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078411.753683:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078411.753686:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078411.753696:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:2.0:1769078411.753701:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5397 00000100:00080000:2.0:1769078411.753709:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078411.753714:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078411.753719:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078411.753742:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616906112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078411.753748:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078411/real 1769078411] req@ffff995b85acbb80 x1855007616905600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078411.753763:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078411.753782:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078411.753784:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078411.753799:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616905600/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078411.753812:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078411.753817:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078411.753820:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616905600:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078411.753841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905728:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078411.753859:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078411.753870:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078411/real 1769078411] req@ffff995b85acaa00 x1855007616905728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078411.753882:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616905728/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078411.753891:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078411.753894:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078411.753897:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905728:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078411.753904:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905856:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078411.753925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078411.753935:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078411/real 1769078411] req@ffff995b85ac9c00 x1855007616905856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078411.753946:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616905856/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078411.753956:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078411.753959:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078411.753961:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905856:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078411.753974:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905984:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078411.753994:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078411.754004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078411/real 1769078411] req@ffff995b85ac8a80 x1855007616905984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078411.754016:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616905984/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078411.754025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078411.754029:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078411.754032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616905984:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078411.754037:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616906112:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078411.754050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078411.754072:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078411/real 1769078411] req@ffff995b85acb800 x1855007616906112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078411.754090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616906112/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078411.754100:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078411.754103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078411.754106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616906112:192.168.204.132@tcp:8:kworker.0 00000100:00080000:2.0:1769078411.754171:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5397 00000100:00080000:2.0:1769078411.754177:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078411.754182:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078411.754188:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078411.754203:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616906240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078411.754285:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078411.754290:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078411.754292:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078411.754297:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5397 00000100:00080000:2.0:1769078411.754302:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078411.754306:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078411.754312:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078411.754324:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616906368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078411.754333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616906240:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078411.754705:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078411.754732:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078411/real 1769078411] req@ffff995b85aca300 x1855007616906240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078411.754751:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616906240/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078411.754762:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078411.754766:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078411.754770:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616906240:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078411.754791:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616906368:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078411.754812:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078411.754823:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078411/real 1769078411] req@ffff995b85acb100 x1855007616906368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078411.754905:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616906368/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078411.754922:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078411.754925:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078411.754929:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616906368:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078412.456625:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078412.456680:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078412.776538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078412.776540:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078412.776577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078412.776582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078413.480613:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078413.481025:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078413.800633:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078413.800823:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078413.800963:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078413.801377:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078414.504590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078414.504642:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078414.824617:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078414.824617:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078414.824716:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078414.824731:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769078415.528623:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078415.528632:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078415.848566:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078415.848577:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078415.848579:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078415.848606:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:1.0:1769078416.552591:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078416.552628:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078416.872571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078416.872571:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078416.872594:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:3.0:1769078416.872622:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078416.872628:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078416.872632:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078416.872633:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078416.872642:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5402 00000100:00080000:3.0:1769078416.872652:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078416.872659:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078416.872691:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559880 x1855007616906496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078416.872721:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078416.872725:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078416.872727:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078416.872731:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5402 00000100:00080000:3.0:1769078416.872734:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078416.872737:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078416.872746:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078416.872757:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616906624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078416.872771:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078416.872774:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078416.872776:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078416.872779:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5402 00000100:00080000:3.0:1769078416.872782:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078416.872785:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078416.872789:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078416.872798:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616906752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078416.872805:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078416.872809:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078416.872811:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078416.872814:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5402 00000100:00080000:3.0:1769078416.872816:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078416.872819:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078416.872824:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078416.872831:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616906880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078416.872838:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078416.872841:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078416.872843:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078416.872846:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5402 00000100:00080000:3.0:1769078416.872850:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078416.872853:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078416.872857:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078416.872864:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616907008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078416.872871:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078416.872874:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078416.872876:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078416.872879:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5402 00000100:00080000:3.0:1769078416.872882:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078416.872885:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078416.872889:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078416.872896:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616907136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078416.872903:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078416.872906:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078416.872908:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078416.872910:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5402 00000100:00080000:3.0:1769078416.872913:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078416.872916:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078416.872920:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078416.872927:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616907264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078416.873005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616906496:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769078416.873067:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078416.873102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078416/real 1769078416] req@ffff995b84559880 x1855007616906496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078416.873117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559880 x1855007616906496/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078416.873126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078416.873129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078416.873133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616906496:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078416.873139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616906624:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078416.873157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078416.873167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078416/real 1769078416] req@ffff995b84558700 x1855007616906624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078416.873177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616906624/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078416.873184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078416.873187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078416.873202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616906624:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078416.873207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616906752:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078416.873223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078416.873233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078416/real 1769078416] req@ffff995b84559c00 x1855007616906752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078416.873243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616906752/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078416.873252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078416.873255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078416.873257:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616906752:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078416.873262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616906880:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078416.873277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078416.873286:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078416/real 1769078416] req@ffff995b8455a300 x1855007616906880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078416.873302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616906880/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078416.873310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078416.873313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078416.873315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616906880:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078416.873320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907008:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078416.873334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078416.873342:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078416/real 1769078416] req@ffff995b8455ad80 x1855007616907008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078416.873351:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616907008/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078416.873359:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078416.873362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078416.873365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907008:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078416.873369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616907136:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078416.873384:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078416.873393:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078416/real 1769078416] req@ffff995b8455b480 x1855007616907136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078416.873403:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616907136/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078416.873415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078416.873418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078416.873421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616907136:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078416.873426:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616907264:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078416.873441:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078416.873450:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078416/real 1769078416] req@ffff995b84558000 x1855007616907264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078416.873499:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616907264/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078416.873507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078416.873510:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078416.873513:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616907264:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078417.576567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078417.576705:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078417.896567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078417.896567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078417.896662:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078417.896664:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078418.600536:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078418.600573:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078418.920563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078418.920563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078418.920563:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078418.920583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078419.624563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078419.627642:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078419.944546:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078419.944546:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078419.944556:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078419.944669:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078420.648549:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078420.648621:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078420.968572:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078420.968574:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078420.968580:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078420.969716:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078421.672569:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078421.672599:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078421.992562:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078421.992562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078421.992563:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078421.992563:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769078421.992602:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078421.992609:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078421.992613:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078421.992622:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5407 00000100:00080000:1.0:1769078421.992632:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078421.992638:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078421.992671:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616907392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078421.992701:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078421.992705:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078421.992706:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078421.992710:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5407 00000100:00080000:1.0:1769078421.992713:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078421.992717:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078421.992725:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078421.992736:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616907520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078421.992748:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078421.992748:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616907392:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078421.992756:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078421.992759:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078421.992763:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5407 00000100:00080000:1.0:1769078421.992765:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078421.992769:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078421.992773:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078421.992783:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616907648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078421.992798:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078421.992801:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078421.992803:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078421.992807:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5407 00000100:00080000:1.0:1769078421.992811:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078421.992814:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000800:00000100:2.0:1769078421.992820:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000008:00100000:1.0:1769078421.992821:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078421.992833:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616907776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078421.992843:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078421.992846:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078421.992847:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078421.992850:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5407 00000100:00080000:1.0:1769078421.992853:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078421.992857:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078421.992861:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078421.992871:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616907904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078421.992876:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078421/real 1769078421] req@ffff995b868c7480 x1855007616907392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078421.992890:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078421.992896:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078421.992898:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078421.992903:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5407 00000100:00100000:2.0:1769078421.992906:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616907392/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078421.992907:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078421.992916:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769078421.992920:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769078421.992921:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1769078421.992926:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078421.992930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616907392:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078421.992935:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616908032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078421.992944:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078421.992947:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078421.992949:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078421.992952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907520:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078421.992957:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5407 00000100:00080000:1.0:1769078421.992962:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078421.992965:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078421.992969:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078421.992982:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616908160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078421.992986:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078421.993006:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078421/real 1769078421] req@ffff995b868c7100 x1855007616907520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078421.993017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616907520/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078421.993023:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078421.993025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078421.993027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907520:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078421.993031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907648:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078421.993044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078421.993050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078421/real 1769078421] req@ffff995b868c5500 x1855007616907648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078421.993058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616907648/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078421.993067:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078421.993070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078421.993073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907648:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078421.993078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907776:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078421.993097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078421.993107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078421/real 1769078421] req@ffff995b868c6300 x1855007616907776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078421.993117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616907776/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078421.993130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078421.993134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078421.993137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907776:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078421.993142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907904:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078421.993157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078421.993166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078421/real 1769078421] req@ffff995b868c5c00 x1855007616907904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078421.993175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616907904/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078421.993183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078421.993185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078421.993196:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616907904:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078421.993201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616908032:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078421.993219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078421.993228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078421/real 1769078421] req@ffff995b868c6a00 x1855007616908032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078421.993237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616908032/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078421.993243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078421.993245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078421.993249:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616908032:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078421.993264:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616908160:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078421.993280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078421.993290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078421/real 1769078421] req@ffff995b868c4a80 x1855007616908160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078421.993300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616908160/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078421.993306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078421.993309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078421.993311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616908160:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078422.698521:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078422.699872:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078423.016516:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078423.016522:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078423.016524:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078423.016564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078423.720554:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078423.720636:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078424.040585:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078424.040678:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078424.040773:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078424.040861:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078424.744558:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078424.744640:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078425.064574:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078425.064574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078425.064575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078425.064700:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078425.768552:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078425.768616:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078426.088571:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078426.088571:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078426.088599:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078426.088706:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078426.792563:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078426.792621:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078427.112535:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769078427.112537:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078427.112538:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078427.112543:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:1.0:1769078427.112546:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078427.112552:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078427.112561:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5412 00000100:00080000:1.0:1769078427.112571:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769078427.112581:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769078427.112585:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078427.112626:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7800 x1855007616908288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078427.112649:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078427.112652:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078427.112654:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078427.112658:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5412 00000100:00080000:1.0:1769078427.112662:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078427.112666:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078427.112680:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616908288:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769078427.112681:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078427.112692:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4700 x1855007616908416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078427.112701:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078427.112703:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078427.112705:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078427.112708:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5412 00000100:00080000:1.0:1769078427.112712:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078427.112715:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078427.112720:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078427.112728:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6d80 x1855007616908544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078427.112733:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078427.112736:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078427.112738:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078427.112740:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078427.112743:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5412 00000100:00080000:1.0:1769078427.112746:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078427.112749:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078427.112753:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078427.112761:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616908672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078427.112767:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078427/real 1769078427] req@ffff995b868c7800 x1855007616908288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078427.112768:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078427.112771:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078427.112773:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078427.112776:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5412 00000100:00080000:1.0:1769078427.112778:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078427.112781:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078427.112783:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7800 x1855007616908288/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078427.112785:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1769078427.112792:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769078427.112793:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616908800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078427.112796:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769078427.112800:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078427.112800:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616908288:192.168.204.132@tcp:250:kworker.0 00000100:00080000:1.0:1769078427.112804:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078427.112805:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078427.112808:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5412 00000100:00080000:1.0:1769078427.112810:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078427.112813:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078427.112817:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078427.112818:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616908416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078427.112824:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616908928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078427.112833:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078427.112836:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1769078427.112837:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078427.112838:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078427.112841:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5412 00000100:00080000:1.0:1769078427.112844:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078427.112847:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078427.112847:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078427/real 1769078427] req@ffff995b868c4700 x1855007616908416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078427.112851:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078427.112859:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616909056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078427.112859:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4700 x1855007616908416/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078427.112868:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078427.112870:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078427.112873:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616908416:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078427.112879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616908544:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078427.112895:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078427.112905:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078427/real 1769078427] req@ffff995b868c6d80 x1855007616908544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078427.112915:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6d80 x1855007616908544/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078427.112928:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078427.112931:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078427.112934:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616908544:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078427.112940:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616908672:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078427.112957:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078427.112969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078427/real 1769078427] req@ffff995b868c4a80 x1855007616908672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078427.112978:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616908672/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078427.112985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078427.112988:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078427.112991:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616908672:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078427.112997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616908800:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078427.113019:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078427.113030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078427/real 1769078427] req@ffff995b868c6a00 x1855007616908800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078427.113040:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616908800/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078427.113048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078427.113051:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078427.113055:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616908800:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078427.113072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616908928:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078427.113093:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078427.113104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078427/real 1769078427] req@ffff995b868c5c00 x1855007616908928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078427.113115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616908928/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078427.113123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078427.113126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078427.113130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616908928:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078427.113138:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616909056:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078427.113158:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078427.113172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078427/real 1769078427] req@ffff995b868c6300 x1855007616909056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078427.113184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616909056/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078427.113192:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078427.113195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078427.113197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616909056:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078427.816519:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078427.816580:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078428.136596:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078428.136596:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078428.136636:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078428.136636:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078428.840604:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078428.840678:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078429.160605:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078429.160605:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078429.160606:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078429.160623:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:1.0:1769078429.864716:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078429.864785:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078430.184629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078430.184629:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078430.184773:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078430.184786:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078430.888531:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078430.888603:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078431.208536:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078431.208536:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078431.208564:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078431.208575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078431.912545:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078431.912735:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078432.232588:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078432.232589:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078432.232589:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:1.0:1769078432.232600:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078432.232605:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078432.232717:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078432.232718:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:1.0:1769078432.232764:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5417 00000100:00080000:1.0:1769078432.232779:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078432.232787:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:1.0:1769078432.232825:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5500 x1855007616909184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078432.232857:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078432.232863:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078432.232867:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078432.232873:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5417 00000100:00080000:1.0:1769078432.232877:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078432.232881:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078432.232892:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616909184:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769078432.232893:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078432.232918:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7100 x1855007616909312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078432.232930:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078432.232935:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078432.232937:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078432.232941:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5417 00000100:00080000:1.0:1769078432.232946:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078432.232950:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078432.232957:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078432.232969:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c7480 x1855007616909440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078432.232980:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769078432.232987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078432.232989:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078432.232992:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078432.232997:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5417 00000100:00080000:1.0:1769078432.233001:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078432.233005:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:1.0:1769078432.233011:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078432.233025:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6300 x1855007616909568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078432.233034:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078432.233039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078432/real 1769078432] req@ffff995b868c5500 x1855007616909184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078432.233040:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078432.233045:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078432.233067:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5417 00000100:00100000:2.0:1769078432.233071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5500 x1855007616909184/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078432.233075:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078432.233090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078432.233091:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00080000:2.0:1769078432.233098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078432.233106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616909184:192.168.204.132@tcp:250:kworker.0 00000008:00100000:1.0:1769078432.233107:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1769078432.233126:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c5c00 x1855007616909696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078432.233137:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078432.233139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616909312:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078432.233152:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078432.233154:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078432.233161:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5417 00000100:00080000:1.0:1769078432.233168:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769078432.233171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078432.233183:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078432.233189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078432/real 1769078432] req@ffff995b868c7100 x1855007616909312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078432.233193:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078432.233240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7100 x1855007616909312/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078432.233242:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c6a00 x1855007616909824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078432.233274:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078432.233280:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00080000:1.0:1769078432.233282:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078432.233287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616909312:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078432.233298:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769078432.233299:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616909440:192.168.204.132@tcp:38:kworker.0 00000100:00080000:1.0:1769078432.233306:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078432.233315:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5417 00000800:00000100:2.0:1769078432.233320:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:1.0:1769078432.233322:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078432.233331:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078432.233332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078432/real 1769078432] req@ffff995b868c7480 x1855007616909440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078432.233348:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078432.233349:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c7480 x1855007616909440/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078432.233365:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078432.233369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078432.233373:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b868c4a80 x1855007616909952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078432.233378:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616909440:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078432.233392:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616909568:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078432.233415:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078432.233424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078432/real 1769078432] req@ffff995b868c6300 x1855007616909568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078432.233433:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6300 x1855007616909568/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078432.233441:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078432.233446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078432.233449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616909568:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078432.233481:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616909696:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078432.233494:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078432.233502:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078432/real 1769078432] req@ffff995b868c5c00 x1855007616909696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078432.233510:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c5c00 x1855007616909696/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078432.233517:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078432.233519:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078432.233521:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616909696:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078432.233526:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616909824:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078432.233536:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078432.233542:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078432/real 1769078432] req@ffff995b868c6a00 x1855007616909824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078432.233549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c6a00 x1855007616909824/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078432.233555:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078432.233557:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078432.233559:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616909824:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078432.233563:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616909952:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078432.233573:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078432.233579:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078432/real 1769078432] req@ffff995b868c4a80 x1855007616909952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078432.233591:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b868c4a80 x1855007616909952/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078432.233596:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078432.233598:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078432.233600:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b868c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616909952:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769078432.936692:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078432.936741:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078433.256542:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078433.256542:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078433.256574:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078433.257500:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078433.960556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078433.960626:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078434.280582:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078434.280582:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078434.280629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078434.280643:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769078434.984713:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078434.985238:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078435.304624:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078435.304625:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078435.304914:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078435.304980:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078436.008611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078436.008799:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078436.328618:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078436.328666:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078436.328757:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078436.330476:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078437.032568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078437.032919:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078437.352620:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078437.352709:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078437.352777:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078437.353025:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078437.353069:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078437.353075:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078437.353078:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078437.353086:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5423 00000100:00080000:3.0:1769078437.353096:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078437.353102:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078437.353156:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616910080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078437.353216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616910080:192.168.204.132@tcp:250:kworker.0 00000800:00000100:2.0:1769078437.353269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00080000:3.0:1769078437.353296:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078437.353300:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078437.353302:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078437.353306:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5423 00000100:00080000:3.0:1769078437.353309:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078437.353313:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078437.353321:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078437.353333:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616910208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078437.353980:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078437/real 1769078437] req@ffff995b84559500 x1855007616910080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078437.354013:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616910080/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078437.354026:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078437.354030:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078437.354034:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616910080:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078437.354057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616910208:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078437.354081:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078437.354097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078437/real 1769078437] req@ffff995b8455aa00 x1855007616910208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078437.354157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616910208/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078437.354166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078437.354169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078437.354173:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616910208:192.168.204.132@tcp:38:kworker.0 00000100:00080000:3.0:1769078437.354313:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078437.354318:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078437.354320:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078437.354324:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5423 00000100:00080000:3.0:1769078437.354329:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078437.354332:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078437.354337:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078437.354349:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b100 x1855007616910336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078437.354359:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078437.354362:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078437.354364:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078437.354367:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5423 00000100:00080000:3.0:1769078437.354370:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078437.354374:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078437.354378:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078437.354385:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a680 x1855007616910464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078437.354394:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078437.354397:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078437.354398:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078437.354401:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5423 00000100:00080000:3.0:1769078437.354403:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078437.354406:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078437.354409:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078437.354417:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558a80 x1855007616910592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078437.354439:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078437.354442:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078437.354443:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078437.354447:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5423 00000100:00080000:3.0:1769078437.354450:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078437.354468:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078437.354472:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078437.354481:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455bb80 x1855007616910720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078437.354495:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078437.354498:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078437.354500:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078437.354503:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5423 00000100:00080000:3.0:1769078437.354506:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078437.354509:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078437.354513:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078437.354521:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455aa00 x1855007616910848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078437.354761:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616910336:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078437.354784:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078437.354795:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078437/real 1769078437] req@ffff995b8455b100 x1855007616910336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078437.354818:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b100 x1855007616910336/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078437.354828:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078437.354831:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078437.354834:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616910336:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078437.354839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616910464:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078437.354858:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078437.354867:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078437/real 1769078437] req@ffff995b8455a680 x1855007616910464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078437.355238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a680 x1855007616910464/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078437.355248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078437.355251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078437.355255:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616910464:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078437.355266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616910592:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078437.355286:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078437.355302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078437/real 1769078437] req@ffff995b84558a80 x1855007616910592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078437.355313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558a80 x1855007616910592/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078437.355325:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078437.355328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078437.355331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616910592:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078437.355336:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616910720:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078437.355353:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078437.355366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078437/real 1769078437] req@ffff995b8455bb80 x1855007616910720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078437.355386:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455bb80 x1855007616910720/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078437.355394:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078437.355396:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078437.355399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616910720:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078437.355407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616910848:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078437.355552:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078437.355569:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078437/real 1769078437] req@ffff995b8455aa00 x1855007616910848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078437.356414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455aa00 x1855007616910848/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078437.356438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078437.356648:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078437.356651:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616910848:192.168.204.132@tcp:38:kworker.0 00010000:00010000:1.0:1769078438.059240:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078438.059411:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078438.376567:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078438.376567:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078438.376567:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078438.376568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078439.080654:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078439.081136:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078439.400523:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078439.400527:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078439.400578:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078439.402225:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078440.104574:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078440.105059:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078440.424577:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078440.424581:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078440.424767:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078440.425161:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00010000:00010000:0.0:1769078441.128720:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078441.128837:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078441.448548:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078441.449900:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078441.452175:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078441.452960:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078442.152600:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078442.153296:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078442.472632:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078442.476187:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078442.476305:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078442.476754:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078442.476761:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078442.476766:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078442.476778:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5428 00000100:00080000:3.0:1769078442.476792:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078442.476800:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:3.0:1769078442.476843:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559500 x1855007616910976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078442.476873:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078442.476877:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078442.476880:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078442.476884:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5428 00000100:00080000:3.0:1769078442.476889:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078442.476892:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078442.476903:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078442.476914:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558000 x1855007616911104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078442.476923:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078442.476927:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078442.476928:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078442.476932:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5428 00000100:00080000:3.0:1769078442.476935:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078442.476938:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078442.476941:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078442.476951:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455b480 x1855007616911232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078442.476960:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078442.476963:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078442.476965:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078442.476969:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5428 00000100:00080000:3.0:1769078442.476972:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078442.476975:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078442.476980:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078442.476990:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455ad80 x1855007616911360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078442.476998:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078442.477001:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078442.477003:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078442.477006:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5428 00000100:00080000:3.0:1769078442.477009:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078442.477012:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078442.477016:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1769078442.477023:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b8455a300 x1855007616911488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078442.477042:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078442.477045:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078442.477047:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078442.477051:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5428 00000100:00080000:3.0:1769078442.477055:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078442.477059:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078442.477064:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078442.477074:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84559c00 x1855007616911616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078442.477082:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616910976:192.168.204.132@tcp:250:kworker.0 00000100:00080000:3.0:1769078442.477086:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078442.477091:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078442.477093:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078442.477098:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5428 00000100:00080000:3.0:1769078442.477101:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078442.477104:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:3.0:1769078442.477110:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078442.477119:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84558700 x1855007616911744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078442.477161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078442.477204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078442/real 1769078442] req@ffff995b84559500 x1855007616910976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078442.477222:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559500 x1855007616910976/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078442.477233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078442.477236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078442.477241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616910976:192.168.204.132@tcp:250:kworker.0 00000100:00100000:2.0:1769078442.477248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616911104:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078442.477267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078442.477277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078442/real 1769078442] req@ffff995b84558000 x1855007616911104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078442.477288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558000 x1855007616911104/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078442.477296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078442.477300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078442.477303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616911104:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078442.477309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616911232:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078442.477323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078442.477331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078442/real 1769078442] req@ffff995b8455b480 x1855007616911232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078442.477347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455b480 x1855007616911232/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078442.477357:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078442.477360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078442.477363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616911232:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078442.477369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616911360:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078442.477391:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078442.477400:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078442/real 1769078442] req@ffff995b8455ad80 x1855007616911360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078442.477411:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455ad80 x1855007616911360/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078442.477420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078442.477423:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078442.477426:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616911360:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078442.477431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616911488:192.168.204.132@tcp:8:kworker.0 00000800:00000100:2.0:1769078442.477446:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:0.0:1769078442.477559:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078442.477586:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078442/real 1769078442] req@ffff995b8455a300 x1855007616911488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078442.477600:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b8455a300 x1855007616911488/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078442.477615:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078442.477618:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078442.477622:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b8455a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616911488:192.168.204.132@tcp:8:kworker.0 00000100:00100000:2.0:1769078442.477663:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616911616:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078442.477683:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078442.477693:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078442/real 1769078442] req@ffff995b84559c00 x1855007616911616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078442.477704:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84559c00 x1855007616911616/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078442.477712:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078442.477715:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078442.477718:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84559c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616911616:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078442.477723:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616911744:192.168.204.132@tcp:38:kworker.0 00000800:00000100:2.0:1769078442.477737:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:2.0:1769078442.477746:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078442/real 1769078442] req@ffff995b84558700 x1855007616911744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078442.477757:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84558700 x1855007616911744/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078442.477764:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078442.477767:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:2.0:1769078442.477771:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84558700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616911744:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769078443.176666:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078443.179120:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078443.496585:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078443.496587:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078443.496728:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078443.496766:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078444.200556:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078444.200619:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078444.520573:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078444.520574:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078444.520739:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078444.521015:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078445.224577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078445.224653:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078445.544591:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078445.544592:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078445.544626:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078445.544655:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078446.248567:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078446.248696:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078446.568597:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078446.568623:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078446.568909:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078446.568926:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00010000:00010000:0.0:1769078447.272694:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078447.272754:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078447.592636:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078447.592812:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078447.593015:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078447.593076:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078447.593083:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078447.593089:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078447.593098:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00080000:2.0:1769078447.593102:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5433 00000100:00080000:2.0:1769078447.593116:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078447.593125:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078447.593172:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9880 x1855007616911872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078447.593205:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078447.593210:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078447.593211:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078447.593216:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5433 00000100:00080000:2.0:1769078447.593219:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078447.593224:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078447.593235:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078447.593246:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8380 x1855007616912000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078447.593257:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078447.593260:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078447.593262:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078447.593266:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5433 00000100:00080000:2.0:1769078447.593269:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078447.593272:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078447.593276:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078447.593289:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616912128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078447.593296:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078447.593299:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078447.593301:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078447.593304:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5433 00000100:00080000:2.0:1769078447.593307:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078447.593310:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078447.593314:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078447.593323:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616912256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078447.593330:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078447.593333:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078447.593335:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078447.593338:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5433 00000100:00080000:2.0:1769078447.593341:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078447.593345:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078447.593348:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078447.593357:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616912384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078447.593365:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078447.593369:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078447.593370:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078447.593374:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5433 00000100:00080000:2.0:1769078447.593377:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078447.593381:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078447.593384:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078447.593393:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616912512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078447.593401:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078447.593405:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078447.593407:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078447.593410:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5433 00000100:00080000:2.0:1769078447.593413:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078447.593416:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078447.593421:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078447.593429:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616912640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078447.593537:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616911872:192.168.204.132@tcp:250:kworker.0 00000800:00000100:1.0:1769078447.593617:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078447.593663:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078447/real 1769078447] req@ffff995b85ac9880 x1855007616911872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078447.593681:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9880 x1855007616911872/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078447.593692:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078447.593696:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078447.593700:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616911872:192.168.204.132@tcp:250:kworker.0 00000100:00100000:1.0:1769078447.593707:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912000:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078447.593730:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078447.593742:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078447/real 1769078447] req@ffff995b85ac8380 x1855007616912000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078447.593754:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8380 x1855007616912000/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078447.593762:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078447.593765:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078447.593768:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912000:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078447.593774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912128:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078447.593791:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078447.593801:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078447/real 1769078447] req@ffff995b85acb100 x1855007616912128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078447.593811:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616912128/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078447.593899:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078447.593904:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078447.593908:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912128:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078447.593915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912256:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078447.593937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078447.593976:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078447/real 1769078447] req@ffff995b85aca300 x1855007616912256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078447.593988:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616912256/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078447.593997:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078447.594001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078447.594004:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912256:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078447.594011:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912384:192.168.204.132@tcp:8:kworker.0 00000800:00000100:1.0:1769078447.594028:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078447.594039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078447/real 1769078447] req@ffff995b85acb800 x1855007616912384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078447.594050:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616912384/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078447.594060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078447.594063:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078447.594066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912384:192.168.204.132@tcp:8:kworker.0 00000100:00100000:1.0:1769078447.594072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616912512:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078447.594090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078447.594100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078447/real 1769078447] req@ffff995b85ac8a80 x1855007616912512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078447.594112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616912512/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078447.594119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078447.594122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078447.594125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616912512:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078447.594131:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616912640:192.168.204.132@tcp:38:kworker.0 00000800:00000100:1.0:1769078447.594148:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:1.0:1769078447.594159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078447/real 1769078447] req@ffff995b85ac9c00 x1855007616912640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078447.594171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616912640/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078447.594179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078447.594183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:1.0:1769078447.594186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616912640:192.168.204.132@tcp:38:kworker.0 00010000:00010000:0.0:1769078448.296695:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078448.298191:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078448.616575:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078448.617662:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078448.619237:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078448.619439:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078449.320577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078449.320677:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078449.640562:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078449.640562:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078449.645082:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078449.645982:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078450.344551:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078450.344650:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078450.664639:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078450.665040:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078450.666062:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078450.674042:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078451.368634:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078451.368675:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078451.688575:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078451.688606:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078451.688767:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078451.688780:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078452.392576:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078452.393326:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078452.712568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078452.713519:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078452.713533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078452.713991:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078452.713997:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078452.714001:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078452.714009:0:68599:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5438 00000100:00080000:2.0:1769078452.714019:0:68599:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078452.714026:0:68599:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:2.0:1769078452.714059:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acaa00 x1855007616912768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078452.714088:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078452.714091:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078452.714093:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078452.714097:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5438 00000100:00080000:2.0:1769078452.714100:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078452.714104:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078452.714112:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078452.714123:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acbb80 x1855007616912896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078452.714131:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078452.714134:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078452.714136:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078452.714139:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5438 00000100:00080000:2.0:1769078452.714142:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078452.714145:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078452.714148:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078452.714158:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac9c00 x1855007616913024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078452.714166:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078452.714168:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078452.714170:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078452.714173:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5438 00000100:00080000:2.0:1769078452.714176:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078452.714179:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078452.714183:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078452.714190:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85ac8a80 x1855007616913152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078452.714198:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078452.714200:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078452.714202:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078452.714205:0:68599:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5438 00000100:00080000:2.0:1769078452.714208:0:68599:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078452.714211:0:68599:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078452.714214:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1769078452.714221:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb800 x1855007616913280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078452.714228:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078452.714231:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078452.714232:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078452.714236:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5438 00000100:00080000:2.0:1769078452.714238:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078452.714242:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078452.714246:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078452.714254:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85aca300 x1855007616913408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078452.714261:0:68599:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078452.714264:0:68599:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078452.714266:0:68599:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078452.714269:0:68599:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5438 00000100:00080000:2.0:1769078452.714272:0:68599:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078452.714275:0:68599:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:2.0:1769078452.714280:0:68599:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078452.714289:0:68599:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b85acb100 x1855007616913536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078452.714377:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616912768:192.168.204.132@tcp:250:kworker.0 00000800:00000100:3.0:1769078452.714437:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078452.714505:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078452/real 1769078452] req@ffff995b85acaa00 x1855007616912768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078452.714522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acaa00 x1855007616912768/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078452.714531:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078452.714535:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078452.714539:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616912768:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078452.714548:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912896:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078452.714579:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078452.714592:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078452/real 1769078452] req@ffff995b85acbb80 x1855007616912896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078452.714603:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acbb80 x1855007616912896/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078452.714611:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078452.714614:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078452.714617:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616912896:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078452.714623:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913024:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078452.714640:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078452.714650:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078452/real 1769078452] req@ffff995b85ac9c00 x1855007616913024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078452.714659:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac9c00 x1855007616913024/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078452.714667:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078452.714670:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078452.714673:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913024:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078452.714677:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913152:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078452.714692:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078452.714699:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078452/real 1769078452] req@ffff995b85ac8a80 x1855007616913152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078452.714709:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85ac8a80 x1855007616913152/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078452.714717:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078452.714719:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078452.714722:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85ac8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913152:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078452.714726:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913280:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078452.714739:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078452.714747:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078452/real 1769078452] req@ffff995b85acb800 x1855007616913280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078452.714756:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb800 x1855007616913280/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078452.714763:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078452.714765:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078452.714768:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913280:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078452.714772:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616913408:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078452.714784:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078452.714791:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078452/real 1769078452] req@ffff995b85aca300 x1855007616913408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078452.714800:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85aca300 x1855007616913408/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078452.714811:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078452.714814:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078452.714816:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85aca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616913408:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078452.714820:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616913536:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078452.714844:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078452.714863:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078452/real 1769078452] req@ffff995b85acb100 x1855007616913536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078452.714873:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b85acb100 x1855007616913536/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078452.714880:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078452.714883:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078452.714885:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b85acb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616913536:192.168.204.132@tcp:38:kworker.0 00000100:00100000:1.0:1769078452.715159:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078453.416555:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078453.418031:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078453.738263:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078453.738391:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078453.738484:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078453.738533:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078454.440592:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078454.440685:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078454.760586:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078454.760604:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078454.760629:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078454.760657:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078455.464559:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078455.464643:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078455.784612:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078455.784642:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078455.784723:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078455.784728:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078456.488580:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078456.490463:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078456.808602:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:0.0:1769078456.809039:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078456.811912:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078456.811913:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078457.512577:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078457.512665:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078457.832586:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078457.832586:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078457.832593:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078457.832859:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078457.832933:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078457.832939:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078457.832943:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078457.832952:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5443 00000100:00080000:0.0:1769078457.832961:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078457.832967:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078457.833001:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84643800 x1855007616913664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078457.833032:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078457.833035:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078457.833037:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078457.833041:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5443 00000100:00080000:0.0:1769078457.833044:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078457.833049:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078457.833058:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078457.833073:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641180 x1855007616913792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078457.833074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616913664:192.168.204.132@tcp:250:kworker.0 00000100:00080000:0.0:1769078457.833089:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078457.833093:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078457.833095:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078457.833099:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5443 00000100:00080000:0.0:1769078457.833104:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078457.833108:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078457.833112:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078457.833121:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641500 x1855007616913920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078457.833129:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078457.833131:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078457.833133:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078457.833136:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5443 00000100:00080000:0.0:1769078457.833140:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078457.833143:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078457.833148:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078457.833157:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616914048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078457.833165:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078457.833168:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078457.833170:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078457.833172:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5443 00000100:00080000:0.0:1769078457.833175:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078457.833178:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078457.833182:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078457.833190:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616914176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078457.833197:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078457.833200:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078457.833201:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078457.833204:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5443 00000100:00080000:0.0:1769078457.833207:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078457.833210:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078457.833214:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078457.833222:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616914304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078457.833236:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078457.833239:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078457.833241:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078457.833244:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5443 00000100:00080000:0.0:1769078457.833247:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078457.833250:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078457.833254:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078457.833264:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616914432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078457.833403:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078457.833442:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078457/real 1769078457] req@ffff995b84643800 x1855007616913664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078457.833479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84643800 x1855007616913664/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078457.833490:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078457.833494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078457.833507:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616913664:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078457.833528:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913792:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078457.833551:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078457.833562:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078457/real 1769078457] req@ffff995b84641180 x1855007616913792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078457.833574:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641180 x1855007616913792/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078457.833583:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078457.833586:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078457.833589:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913792:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078457.833595:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913920:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078457.833613:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078457.833623:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078457/real 1769078457] req@ffff995b84641500 x1855007616913920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078457.833634:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641500 x1855007616913920/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078457.833644:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078457.833647:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078457.833650:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616913920:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078457.833655:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914048:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078457.833673:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078457.833683:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078457/real 1769078457] req@ffff995b84640e00 x1855007616914048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078457.833694:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616914048/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078457.833715:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078457.833719:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078457.833722:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914048:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078457.833727:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914176:192.168.204.132@tcp:8:kworker.0 00000800:00000100:3.0:1769078457.833744:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078457.834393:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078457/real 1769078457] req@ffff995b84642680 x1855007616914176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078457.834406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616914176/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078457.834416:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078457.834419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078457.834422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914176:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078457.834428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616914304:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078457.834447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078457.834474:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078457/real 1769078457] req@ffff995b84640000 x1855007616914304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078457.834486:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616914304/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078457.834494:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078457.834559:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078457.834563:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616914304:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078457.834569:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616914432:192.168.204.132@tcp:38:kworker.0 00000800:00000100:3.0:1769078457.834590:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.132@tcp 00000100:00100000:3.0:1769078457.834602:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078457/real 1769078457] req@ffff995b84640380 x1855007616914432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078457.834613:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616914432/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078457.834621:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078457.834624:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078457.834627:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616914432:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078458.536568:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078458.536914:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078458.856611:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078458.856612:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078458.856682:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078458.857626:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078459.560611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078459.560875:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078459.880568:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078459.880573:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078459.880578:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078459.880694:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078460.584573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078460.584659:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078460.904629:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:1.0:1769078460.904638:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078460.904681:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078460.904713:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078461.608603:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078461.608702:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078461.928685:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:3.0:1769078461.928864:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078461.928950:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:2.0:1769078461.929020:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:2.0:1769078462.632611:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078462.632970:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078462.952658:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078462.952660:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00080000:0.0:1769078462.952670:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) c913fc96-1499-4a5b-8723-2b5b0deb123f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078462.952814:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078462.952821:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078462.952823:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00080000:0.0:1769078462.952832:0:55757:0:(import.c:509:import_select_connection()) MGC192.168.204.132@tcp: connect to NID 192.168.204.132@tcp last attempt 5448 00000100:00080000:0.0:1769078462.952843:0:55757:0:(import.c:588:import_select_connection()) MGC192.168.204.132@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078462.952845:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078462.952852:0:55757:0:(import.c:623:import_select_connection()) MGC192.168.204.132@tcp: import ffff995b82b68000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000100:00100000:0.0:1769078462.952891:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640a80 x1855007616914560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078462.952920:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078462.952924:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078462.952926:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078462.952930:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5448 00000100:00080000:0.0:1769078462.952934:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078462.952938:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b85db1000: import ffff995b8522a000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078462.952948:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078462.952961:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641f80 x1855007616914688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078462.952972:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078462.952975:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078462.952976:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078462.952980:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5448 00000100:00080000:0.0:1769078462.952983:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078462.952986:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b85db1000: import ffff995b8522b000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078462.952990:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078462.953071:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84641c00 x1855007616914816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078462.953082:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078462.953084:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078462.953086:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078462.953090:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5448 00000100:00080000:0.0:1769078462.953094:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078462.953097:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff995b85db1000: import ffff995b8522d000 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078462.953102:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078462.953115:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640380 x1855007616914944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078462.953123:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 0e3d52fd-a30d-4263-b65e-2d3fbcbbec40->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078462.953126:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078462.953127:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078462.953130:0:55757:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: connect to NID 192.168.204.132@tcp last attempt 5448 00000100:00080000:0.0:1769078462.953133:0:55757:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078462.953137:0:55757:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff995b85db1000: import ffff995b8522d800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078462.953141:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681348 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1769078462.953150:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640000 x1855007616915072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078462.953158:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078462.953161:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078462.953163:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078462.953166:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5448 00000100:00080000:0.0:1769078462.953169:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078462.953173:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff995b87ecd000: import ffff995b87d95800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078462.953177:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078462.953186:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84642680 x1855007616915200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078462.953194:0:55757:0:(pinger.c:229:ptlrpc_pinger_process_import()) 55248ee8-1097-44b2-9a8f-4dfaa937cd5d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078462.953197:0:55757:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078462.953199:0:55757:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078462.953203:0:55757:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: connect to NID 192.168.204.132@tcp last attempt 5448 00000100:00080000:0.0:1769078462.953205:0:55757:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078462.953208:0:55757:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff995b87ecd000: import ffff995b84428800 using connection 192.168.204.132@tcp/192.168.204.132@tcp 00000008:00100000:0.0:1769078462.953212:0:55757:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681348 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078462.953220:0:55757:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff995b84640e00 x1855007616915328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078462.966895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616914560:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078462.967111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078462.967143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078462.967165:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078462.967182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616915072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078462.967203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616915200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078462.967236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616915328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078463.656590:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078463.656652:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078463.976601:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078463.976725:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:1.0:1769078463.976898:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078463.977909:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078464.680573:0:13603:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078464.681043:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078465.000566:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769078465.000607:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:1.0:1769078465.000769:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:3.0:1769078465.001549:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00010000:00010000:0.0:1769078465.704622:0:13582:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078465.704680:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1769078466.024554:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078466.024582:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:0.0:1769078466.024583:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:1.0:1769078466.024710:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000400:00000100:2.0:1769078466.024778:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.132:988 00000400:00000100:2.0:1769078466.024814:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.132@tcp at host 192.168.204.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078466.024829:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078466.024863:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078466.024872:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078466.024879:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078466.024885:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078466.024892:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000800:00000100:2.0:1769078466.024904:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.132@tcp 00000100:00100000:3.0:1769078466.024934:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078462/real 1769078466] req@ffff995b84640a80 x1855007616914560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078466.024957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640a80 x1855007616914560/t0(0) o250->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078466.024967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b82b68000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078466.024972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078466.024976:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c913fc96-1499-4a5b-8723-2b5b0deb123f:2416:1855007616914560:192.168.204.132@tcp:250:kworker.0 00000100:00100000:3.0:1769078466.024983:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078462/real 1769078466] req@ffff995b84641f80 x1855007616914688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078466.024994:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641f80 x1855007616914688/t0(0) o38->lustre-MDT0000-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078466.025007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078466.025010:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078466.025013:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914688:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078466.025019:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078462/real 1769078466] req@ffff995b84641c00 x1855007616914816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078466.025029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84641c00 x1855007616914816/t0(0) o38->lustre-MDT0001-mdc-ffff995b85db1000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078466.025038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522b000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078466.025041:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078466.025044:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914816:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078466.025050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078462/real 1769078466] req@ffff995b84640380 x1855007616914944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078466.025059:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640380 x1855007616914944/t0(0) o8->lustre-OST0000-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078466.025068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078466.025071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078466.025074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616914944:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078466.025080:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078462/real 1769078466] req@ffff995b84640000 x1855007616915072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078466.025090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640000 x1855007616915072/t0(0) o8->lustre-OST0001-osc-ffff995b85db1000@192.168.204.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078466.025099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b8522d800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078466.025102:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078466.025105:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:0e3d52fd-a30d-4263-b65e-2d3fbcbbec40:2416:1855007616915072:192.168.204.132@tcp:8:kworker.0 00000100:00100000:3.0:1769078466.025111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078462/real 1769078466] req@ffff995b84642680 x1855007616915200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078466.025120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84642680 x1855007616915200/t0(0) o38->lustre-MDT0000-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078466.025128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b87d95800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078466.025131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078466.025134:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616915200:192.168.204.132@tcp:38:kworker.0 00000100:00100000:3.0:1769078466.025139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078462/real 1769078466] req@ffff995b84640e00 x1855007616915328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078466.027701:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff995b84640e00 x1855007616915328/t0(0) o38->lustre-MDT0001-mdc-ffff995b87ecd000@192.168.204.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078466.027716:0:2416:0:(import.c:65:import_set_state_nolock()) ffff995b84428800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078466.027721:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.132@tcp failed (-110) 00000100:00100000:3.0:1769078466.027725:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff995b84640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:55248ee8-1097-44b2-9a8f-4dfaa937cd5d:2416:1855007616915328:192.168.204.132@tcp:38:kworker.0 00000100:00100000:2.0:1769078466.728566:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078466.728686:0:68227:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078467.048610:0:68154:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83f013c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769078467.049124:0:68153:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b8351e480 going to sleep for 0 seconds 00000100:00100000:3.0:1769078467.051182:0:68155:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b90d67d80 going to sleep for 0 seconds 00000100:00100000:0.0:1769078467.051254:0:68152:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b83342300 going to sleep for 0 seconds 00000100:00100000:2.0:1769078467.752573:0:6176:0:(client.c:2690:ptlrpc_set_wait()) set ffff995b9300a9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078467.752837:0:6177:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 81198 lines, 81198 kept, 0 dropped, 0 bad.